2013

Return to search form  

This presentation is one of 200 to be presented during the poster session.
Poster 203: Using Self-Determination Theory to Evaluate Bystander Intervention Programs
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Paula M Adams, Washington State University, padams@wsu.edu
Gitanjali Shrestha, Washington State University, gitanjali.shrestha@email.wsu.edu
Stephanie Roeter, Washington State University, stephanie.roeter@email.wsu.edu
Annelise Smith, Washington State University, annelise.smith@email.wsu.edu
Laura Hill, Washington State University, laurahill@wsu.edu
Abstract: Bystander intervention programs have been suggested as an effective method to reduce interpersonal violence (IPV) (e.g., sexual assault, stalking, dating/domestic violence), but research is limited about effective program components and evaluation methods. Self-Determination Theory (SDT) is a theory of motivation that has been used to explain multiple health-related behaviors, including bystander intervention behaviors. The purpose of this project was to test the usability of an SDT-based evaluation tool that could potentially be used to assess the likelihood of any bystander intervention program resulting in behavior change, regardless of program components. Measures assessing core SDT constructs of motivation, perceived self-competence, and connection to others were adapted into a short, easily-used tool and administered both before and three months after participation in a bystander intervention training. Results indicated the tool was easy to use and may be a reliable predictor of behavior change.
Poster 84: Transforming Organizational Performance: A Model for Evaluating the Effectiveness of a CTSA
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Angela Alexander, University of California at San Diego, a1alexander@ucsd.edu
Sonal Desai, University of California at San Diego, srdesai@ucsd.edu
John Fontanesi, University of California at San Diego, jfontanesi@ucsd.edu
Doris Dworschak, University of California at San Diego, ddworschak@ucsd.edu
Daniel Bouland, University of California at San Diego, dbouland@ucsd.edu
Abstract: The University of California San Diego (UCSD), Clinical and Translational Research Institute (CTRI) is customizing a Balanced Scorecard (BSC) to align business activities with its vision and strategy, improve internal and external communication, and monitor organizational performance against strategic goals. The data obtained from the BSC will provide specific information to direct meaningful change. The goal of the BSC is to: 1. Establish a communication forum for leadership to align strategy and priorities 2. Create a common vision for the organization's mission 3. Implement a uniform strategic planning framework and measurement tool 4. Outline a measurement strategy, using innovative approaches such as Data Envelopment Analysis (DEA), which informs change and enables management to make decisions that optimize enterprise performance This poster will demonstrate how the UCSD CTRI has implemented the BSC and what lessons have been learned from the tool's implementation within an academic research environment.
Poster 201: Evaluating Whether an Organizational Committee Is Working or Not: Lessons Learned
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Peggy Alexiadis Brown, Dalhousie University, palexiad@dal.ca
Abstract: Evaluation of organizational effectiveness as an internal evaluator can be challenging. However requests to evaluate governance structures and/or committee activities within organizations can provide a unique opportunity to build capacity in program evaluation. There are a number of frameworks that can be used to evaluate committee effectiveness and efficiencies. This poster will present a few of these frameworks as well as the processes that were used in a recent evaluation of a senior management group. While no actual data from the evaluation can be shared, lessons learned regarding the process, from development, implementation to sharing, dissemination and utilization of results will be shared.
Poster 26: Social Media, Citizen Science, Crowd-Sourcing, QR Codes, Crowd-Funding, User Generated 'Big Data': Is There a Role for 21st Century Tools in Evaluation Science?
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Aaron Alford, Battelle Memorial Institute, alforda@battelle.org
James Derzon, Battelle Memorial Institute, derzonj@battelle.org
Rich Ann Baetz, Battelle Memorial Institute, rabaetz@gmail.com
Melanie Chansky, Battelle Memorial Institute, chanskym@battelle.org
Abstract: In the past 13 years, advanced networks and computing systems have allowed the creation and/or expansion of unique forms of human interaction and social organization at an unprecedented scale. Social media, citizen science platforms, crowdsourcing, crowdfunding, QR codes and a variety of mass collaboration platforms allow researchers unprecedented access to the thoughts, opinions, movements, private data, labor and resources of tens of millions of people. These tools have been used to great advantage in many sciences ranging from paleontology and astronomy to computer science and bacteriology. This presentation will explore some novel uses of these tools in sciences other than evaluation science as a warm-up and introduction to advanced applications for research. Breakout groups will consider general questions concerning the potential for using these tools in evaluation, as well as three specific themes: 1) Ethics, 2) How and why, 3) Map an application in your area of expertise.
Poster 19: Evaluation of Continuing Medical Education/Medical Education Programs Using the Kirkpatrick's Evaluation of Training Programs and the Continuous Quality Improvement (CQI) Circle
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Asma Ali, American Society for Clinical Pathology, asma.ali1@gmail.com
Eric Parks, American Society for Clinical Pathology, eric.parks@ascp.org
Abstract: This proposed workshop focuses on demonstrating practical tools and strategies for implementing program evaluation of Continuing Medical Education/Continuing Education activities. In order to maintain accreditation, the American Council on Continuing Medical Education (ACCME) "expects providers to take an active, ongoing role in maintaining their compliance with accreditation requirements and improving the CME programs." However, the process for implementing these expectations is often unclear to CME/CE providers because current AACME standards provide limited information on practical tools for implementing these requirements. In this presentation, we provide a demonstration of the American Society for Clinical Pathology (ASCP) method for program evaluation and continuous quality improvement plan for CME/CE. This method has been successfully implemented since 2010, and utilizes a six-step process that focuses on data-driven decision-making, strategic planning, and continuous program improvement. Using a combination of didactic learning, case studies, and discussion, ASCP staff will demonstrate a concrete process to ensure compliance with ACCME medical education requirement based on Kirkpatrick's model for program evaluation and the "Plan, Study, Do" CQI Circle to address the ACCME requirement for program improvement. Case study examples based on ASCP's education programs for pathologists and laboratory professionals will be detailed. In addition, practical tools and strategies such as performance dashboards and scorecards that can facilitate compliance with CE/CME requirements.
Poster 133: Mapping of Investment Against Post Crisis Needs Assessment (PCNA) Recommendations Through Government of Pakistan and Donors Agencies
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Ahmed Ali Khattak, Planning and Development Department FATA Secretariat, ahmedali50@hotmail.com
Abstract: As part of ten (10) years governance program preparation mapping of foreign and government's investment against the recommendations of Post Crisis Needs Assessment is being undertaken. It includes an in dept analysis of Annual Development Plan, Public Sector Development Plan, People Works Program and Foreign Investments by the donors through Government or directly by the donors through their implementing partners. Following are the main objectives of this important exercise. - To highlight and quantify level of efforts FATA secretariat is chipping in for development of various sectors over last five years - To carry out mapping of all the invests from 2010-11 to 2011-12 through ADP, PSDP, PWP and donors for determining a pattern of investment in FATA corresponding to the four strategic objectives and recommendations of the PCNA. The paper will have two main parts: 1. Donor and Government Investments against PCNA 2. Strategy for addressing the Gaps
Poster 174: The Development of a Process Evaluation Plan for a Family-based, Weight Loss Intervention for African American Adolescents
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Kassandra A Alia, University of South Carolina, kuglerk@email.sc.edu
Dawn K Wilson, University of South Carolina, profdwilson@hotmail.com
Sara M St George, University of South Carolina, sara.m.stgeorge@gmail.com
VaShawn Heatley, University of South Carolina, heatleyv12@students.ecu.edu
Rachel Davis, University of South Carolina, rdavis@mailbox.sc.edu
Heather Kitzman-Ulrich, University of North Texas, heather.kitzman-ulrich@unthsc.edu
Tyler McDaniel, Georgia Southern University, tm02713@georgiasouthern.edu
Ruth Saunders, University of South Carolina, rsaunder@mailbox.sc.edu
Abstract: Process evaluation may lead to more effective weight loss interventions for youth by informing program implementation. This presentation will describe the process evaluation of Project Families Improving Together (FIT) for weight loss, an efficacy trial testing a motivational plus family weight loss program on reducing weight in African American adolescents. Social Cognitive, Family Systems and Self-Determination theories guided the development of essential elements for creating a positive group climate. A pilot study demonstrated high fidelity ratings for facilitator implementation of essential elements (behavioral skills=3.88-¦0.36; communication=3.97-¦0.17; autonomy support=3.96-¦0.21; social support=3.69-¦0.47; all ratings 1-4pt scale) and acceptable dose of program components (GëÑ75% adherence). The purpose of this presentation is to demonstrate how results from the pilot study were used to expand the FIT theoretical model to include elements across individual, facilitator, and group ecological levels. Expansion of the conceptual model to multiple systems allows for a comprehensive evaluation of program implementation.
Poster 227: A Meta-analysis of Studies of the Effect of Immersion Programs on Math Achievement in Elementary Schools
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Saeed Almueed, Western Michigan University, sma5454@hotmail.com
Abstract: Lately there has been a call in the profession for more research investigating how foreign language study benefits elementary and secondary students. During the last 20 years, the links between language competence and mathematics learning have become an important area of research. In particular, issues of learning mathematics in multilingual contexts have attracted attention. The current meta-analysis is intended to contribute to the previous studies done, which examines the effectiveness of elementary foreign language immersion programs. A random effect model was used as the computational model. The findings of this meta-analysis of studies indicate a positive effect of 0.361 in the favor of foreign language immersion programs' impact in the math achievement in elementary schools. Conclusion indicates that bilingual education programs are effective in promoting academic achievement, and that sound educational policy should permit and even encourage the development and implementation of bilingual education programs.
Poster 242: Overweight Prevention Project
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Jane Anderson, University of Wisconsin at Stout, andersonjan@my.uwstout.edu
Allison Hinman, University of Wisconsin at Stout, hinmana@my.uwstout.edu
Nicole  Einerson, University of Wisconsin at Stout, einersonn0143@my.uwstout.edu
Abstract: Childhood obesity is becoming an increasing concern for individuals, families, communities and the nation. Research has shown childhood obesity can have negative consequences on quality of life. A review of relevant literature guided our needs assessment to focus on environmental factors and determinants that impact childrens eating behaviors and levels of physical activity. 3rd-5th graders, their parents and teachers from a Midwestern town completed surveys assessing eating and physical activity behavior. Specifically, questions were asked regarding personal levels of physical activity and eating habits. To assess determinants, children, parents and teachers also responded to questions assessing their attitudes, levels of self-efficacy and knowledge, perceived barriers and supports, and outcome expectations toward healthy eating and exercise. The findings of this needs assessment will form a base of knowledge that will inform future interventions as well as increase awareness in children, parents and teachers about factors that contribute to childhood obesity.
Poster 93: Louisiana Department of Health and Hospitals Tobacco Control Program: Schools Putting Prevention to Work, 1-Year Follow-Up Evaluation
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Dodie Arnold, Pennington Biomedical Research Center, dodie.arnold@pbrc.edu
Elizabeth Gollub, Pennington Biomedical Research Center, elizabeth.gollub@pbrc.edu
Brandi Bourgeois, Louisiana Department of Health and Hospitals, brandi.bourgeois@la.gov
Abstract: The Louisiana Department of Health and Hospitals Tobacco Control Program, Schools Putting Prevention to Work (SPPW) Program was a 1-year pilot program designed to assist 27 of Louisiana's 69 school districts in developing comprehensive wellness policies and creating healthier school environments. The wellness policy was to cover 100% tobacco-free campuses, healthier food choices, and increased opportunities for physical activity. SPPW school districts received trainings and on-going technical assistance as well as a $17,000 mini-grant to support their effort. SPPW required that school districts implement a series of local activities to engage school communities and encourage sustainability of healthier behaviors. This follow-up evaluation utilized a survey approach to review the wellness accomplishments and challenges experienced by the SPPW school districts during the post-pilot year (January 2012-December 2012).
Poster 94: Louisiana Tobacco Control Program: Mid-point Evaluation of the Tobacco Free Healthcare Partnership Project
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Dodie Arnold, Pennington Biomedical Research Center, dodie.arnold@pbrc.edu
Brandi Bourgeois, Louisiana Department of Health and Hospitals, brandi.bourgeois@la.gov
René Stansbury, Southwest Louisiana Area Health Education Center, tobaccocontrol@swlahec.com
Abstract: The Tobacco Free Healthcare Partnership Program (TFHPP) is a new and exciting strategy. Traditionally, the State of Louisiana implemented tobacco control programs by support to community-based non-profit organizations. The TFHPP is a change in strategy, a unique program designed to implement comprehensive tobacco control programs in 4 selected hospital systems and the regions they serve within the State of Louisiana. This multi-level program focuses on tobacco control and prevention, cessation programs, and eliminating non-smokers exposure to secondhand smoke through policy change, media efforts, and cessation services. This mid-point evaluation was conducted in order to assess our progress to-date, determine programmatic strengths, and identifies areas for improvement as we enter the final year of the program and transition to issues of sustainability and replication. This evaluation serves as a valuable case study for other states as they consider supplemental/alternative strategies to their current tobacco control efforts.
Poster 202: Diversity in the 21st Century: Infusing Culturally Responsive Techniques Into Evaluation
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Kshawna Askew, University of North Carolina at Greensboro, kcaskew@gmail.com
Holly Downs, University of North Carolina at Greensboro, hadowns@uncg.edu
Abstract: As a result of rapid demographic changes and the increasing multicultural nature of the United States, evaluators are faced with attending to the diversity of the 21st century. One of the implications of diversity is cultural competence. The urgency of cultural competence is apparent in the field of evaluation with the modification of AEA Standards and increased attention toward Culturally Responsive Evaluation (CRE). With the growing awareness of CRE, evaluators desire to be more competent about the culturally responsive approach to evaluation. This poster will provide an overview of CRE, a comparison of culturally responsive and culturally unresponsive approaches using common evaluation methods, and an outline of the implications associated with modifying common evaluation methods to align with CRE.
Poster 245: Innovative Methods for the Use of Case Studies in Evaluations Involving Marginalised Populations
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Rachel Aston, University of Melbourne, rachel.aston@unimelb.edu.au
Abstract: Marginalised populations often stand to benefit the most from evaluations; however, often they are also the most difficult to engage in evaluation. Further barriers to these evaluations include tight budgets and inadequate levels of data. It is therefore important to consider the personal narrative of participants as an effective method to capture the voice of these populations, empowering the participants and increasing the validity of the findings. The evaluation was of an alternative model of education for youth who were disengaged from mainstream secondary schooling. Data was gathered from interviews, student surveys, term reports, and work and communication samples. Utilising inductive methods the data was sorted into four categories and a narrative was built from this. The key outcomes of this methodology include the ability to demonstrate the individual journey of participants and inform future program outcomes.
Poster 12: Building Cross-agency Longitudinal Datasets From the Ground Up: Obtaining, Merging, and Preparing Data to Answer Research Questions
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Elizabeth Autio, Education Northwest, elizabeth.autio@educationnorthwest.org
Sarah Frazelle, Education Northwest, sarah.frazelle@educationnorthwest.org
Jared Silver, Harvard University, jared_silver@gse.harvard.edu
Havala Hanson, Education Northwest, havala.hanson@educationnorthwest.org
Abstract: Data is everywhere. As researchers, our task is to transform data into information that answers questions and guides better decision-making. However, existing data is often collected for purposes that do not align with research use, siloed in separate agencies, and protected by different privacy laws. The purpose of this presentation is to improve researchers' ability to construct longitudinal, cross-agency datasets where none exist in order to answer important policy and research questions. Contributors in this session will discuss the challenges of creating high-quality longitudinal datasets that include data from multiple agencies. They will suggest strategies for obtaining and securing personally identifiable information, merging datasets without common identifiers, and overcoming common data quality issues with longitudinal datasets.
Poster 1: Developing and Using a Fidelity of Implementation Index to Describe Complex Educational Reforms
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Laureen Avery, University of California at Los Angeles, avery@gseis.ucla.edu
Jason Cervone, University of California at Los Angeles, jcervone@gseis.ucla.edu
Lisa DiMartino, University of California at Los Angeles, dimartino@gseis.ucla.edu
Chris Campbell, Plymouth Public Schools, ccampbell@plymouth.k12.ma.us
Sean Halpin, Plymouth Public Schools, shalpin@plymouth.k12.ma.us
Abstract: The New England Network for Personalization and Performance (the NETWORK) is made up of thirteen high schools across four New England states. The project hypothesis is: A network of schools, working together to create authentic tasks and common rubrics to measure uncommon assessment tasks, will foster personalized learning resulting in higher student achievement, as demonstrated by lower dropout rates, higher graduation rates, and demonstrable success after high school. The project, funded by the USDOE's Investing in Innovation program, is being evaluated using a quasi-experimental design with a matched comparison group. Evaluators have developed and implemented a Fidelity of Implementation index to capture and describe 'the project'. This task is particularly challenging in a development project, since the activities and elements are refined and shifted from year to year. Ultimately the act of defining and measuring has promoted project improvement as the relationships between activities and outputs are continually examined.
Poster 243: Examining Local Level Surveys for Larger Scale Evaluative Use: A Factor Analysis of a Student Survey on Mathematics Attitudes
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Erika Baldwin, University of California at Santa Barbara, ebaldwin@education.ucsb.edu
Abstract: As evaluation grows as more of a necessity in educational research, the need to establish validated instruments and measures becomes more evident. These instruments can contribute to the field of educational evaluation by providing a more efficient and effective mechanism for assessing students attitudes and feelings. This study examines the factor structure, validity and reliability of a student survey used in the evaluation of the ongoing California Academic Partnership Program (CAPP) Algebra & Formative Assessments Project. This study aims to develop a local-level survey metric that can be used more globally in order to provide for more efficient evaluation methods, as well as investigate the realm of research surrounding small-scale instruments and measures and the importance of examining the quality of these tools.
Poster 183: Predicting Attendance at After School Programs: The Role of Individual Student and Site Level Variables
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Alyssa Banford, University of Connecticut, banford@gmail.com
Hannah Mudrick, University of Connecticut, hannah.mudrick@uconn.edu
Sara Johnson, Tufts University, sarakassiejohnson@gmail.com
Stephen Anderson, University of Connecticut, stephen.anderson@uconn.edu
Abstract: Concerns about the recruitment and retention of students in after school programs exist nationwide. The goal of programs to improve students' academic and social success cannot be achieved without regular attendance. These concerns are even more apparent in programs that target older and hard-to-reach youth, those who could especially benefit from the opportunities afforded through after school programming. The current study used data from 12,495 students from 146 after school programs operating in Connecticut during the 2011-12 school year to evaluate student and site level predictors of participants' attendance rates at after school programs. Data were drawn from weekly attendance records, the Connecticut State Department of Education, and an End of Year Report completed by site coordinators. This evaluation is particularly valuable because it adds to the knowledge base about how programs can recruit students and promote higher levels of attendance, a fairly understudied area in the after school literature.
Poster 193: Participatory Action Evaluation - Emergent method of Evaluation
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Cindy Banyai, Refocus Institute, cindy.banyai@gmail.com
Abstract: Participatory action evaluation (PAE) applies participatory action research techniques and empowerment evaluation in the pursuit of evaluative data and capacity building. PAE uses non-traditional media such as participatory photography, participatory video, metaphor drawing, dramatic interpretation, or collaborative art in group projects with an evaluative objective. This qualitative approach to evaluation is a collaborative inquiry process implemented through the activities of a group and public exhibition. The poster illustrates PAE implementation steps, highlights cases employing the method, and makes reference to the fundamental principles guiding the approach.
Poster 232: Strategies for Organizing a Nonprofit to Effectively MeasureaAnd Evaluate its Impact
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Eric Barela, Partners in School Innovation, ebarela@partnersinschools.org
Dana McCurdy, Partners in School Innovation, dmccurdy@partnersinschools.org
Abstract: Many nonprofits are currently conducting impact evaluations to determine their contributions to improved conditions for program participants. However, the process of organizing to effectively evaluate impact can be difficult to navigate, especially as a nonprofit undertakes the process of scaling up. This is a case study of one school-reform nonprofit's internal efforts to measure and evaluate its impact. These efforts included restructuring the evaluation arm of the nonprofit and shifting an organizational mindset around measuring impact, rather than simply measuring performance. The added challenge of organizing to measure and evaluate impact while scaling up will also be discussed. Both successful and unsuccessful strategies will be shared as a way to show the range of effort needed to organize around effective measurement and evaluation of impact.
Poster 244: Measuring Parent Perceptions of Climate Factors - Are We Measuring What We Think We are Measuring?
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Michael Barnes, Columbus City Schools, mrbarnes@mac.com
Abstract: Parents in a large urban school district are annually surveyed about school climate. A factor analysis revealed one factor measured by this instrument. A different instrument was selected this year. Purposeful administration and analysis will be done to try to answer the following questions. Does the observed factor structure confirm the proposed six-factor structure of the instrument? Do the translated versions measure the same factors? Is the survey to long? Are the respondents representative of the school populations? The English version will be administered in a full length and half-length versions randomly distributed in equal proportions at some schools. Somali and Spanish versions will be distributed to students in households where those languages are spoken. At some schools a randomly selected sample will be targeted before mass distribution to the school population. The full results as well as the various experimental administrations will be analyzed to a nswer these questions.
Poster 51: Effective Strategies to Address Diabetes Related Disparities
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Marilyn Batan, Centers for Disease Control and Prevention, fpl8@cdc.gov
Bina Jayapaul-Philip, Centers for Disease Control and Prevention, ify3@cdc.gov
Stephanie Rutledge, Centers for Disease Control and Prevention, sbr4@cdc.gov
Abstract: We will describe the process that CDC's Division of Diabetes Translation, along with the Research Triangle Institute, used to assess the effectiveness of evidence-based interventions that several states have implemented to reduce diabetes complications. These interventions focus on diabetes self-management education and quality improvement in health systems. Our team conducted a non-systematic literature review. We identified key themes in the barriers and challenges faced by groups suffering from health disparities, such as low income, rural and ethnic minority populations. We used these identified themes to revise an existing conceptual model (Reach, Effectiveness, Adoption, Implementation, and Maintenance [RE-AIM]) framework and interview protocol. We then interviewed programs implementing interventions aimed at reducing diabetes related disparities. A report, highlighting disparities issues within the modified RE-AIM framework and including several case studies, is in preparation.
Poster 46: Evaluating a Spanish Basic Language Program at a Mid-sized Southeastern University
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Holli Bayonas, iEvaluate LLC, holli@i-evaluate.net
Mariche García-Bayonas, University of North Carolina at Greensboro, megarcia@uncg.edu
Abstract: The author and Director of Language Instruction at a mid-sized public university are designing an evaluation of the Basic Language Program in Spanish. The Director has engaged in informal evaluation of the program since assuming the position in Fall 2009, but within the contextual factors of the university, such as adhering to the regional accreditation requirements, focusing on student learning outcomes, and the traditional program review process led by the university's Institutional Research office, and office of the Chancellor. The presentation will document the steps involved in designing the evaluation, the design, and some preliminary results. Of particular interest will be the integration of the Student Learning Outcomes within the logic model.
Poster 66: When Observation is Not an Option: The Self-Study Method of Assessing Fidelity of Implementation in College Readiness Programs
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Tiffany Berry, Claremont Graduate University, tiffany.berry@cgu.edu
Michelle Sloper, Claremont Graduate University, michelle.sloper@cgu.edu
Samantha Langan, Claremont Graduate University, samantha.langan@cgu.edu
Abstract: Evaluation paradigms urge evaluators to assess fidelity of implementation (FOI) to clarify relationships between program activities and outcomes, contribute to organizational improvement, and enhance the generalizability of evaluation findings (Chen, 1990; Century, Rudnick & Freeman, 2010; Dane & Schneider, 1998). Site observations are considered one of the best methods to measure FOI. However, in sensitive contexts where observations are intrusive, evaluators have few alternative tools available to measure FOI. The purpose of this presentation is to introduce the self-study methodology, an innovative FOI approach we developed that includes (1) determining criteria for effectiveness, (2) engaging multiple stakeholders to rate their level of implementation, and (3) triangulating among stakeholders to identify areas of agreement/disagreement. The authors will share their experiences implementing this methodology in the evaluation of Project GRAD Los Angeles (a multi-site college readiness program) as well as discuss practical guidelines for applying the self-study in diverse evaluation contexts.
Poster 72: Addressing Water Quality Mitigation Challenges Through Evaluation
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Troy Berthold, Texas A&M University, taberthold@ag.tamu.edu
Abstract: Agricultural watershed programs are implemented by: 1) identifying an impaired water body, 2) educating producers on both the impairment and strategies for mitigation, 3) implementing sustainable agricultural practices, and 4) monitoring for success. Not taken into consideration are the actual educational needs of producers, barriers to practice adoption, and producers' perception of watershed implementation programs. Two of these factors should be evaluated prior to implementing watershed projects to help target and address the needs of agricultural producers and better encourage the adoption of voluntary water quality mitigation practices. Additionally, an overall program evaluation should occur post implementation to help managers strategize the overall approach and tailor the program according to results. This type of evaluation is lacking in todays watershed implementation programs and should be utilized to efficiently plan, prioritize efforts, use resources efficiently, and help managers positively impact water quality.
Poster 141: Engaging New Program Staff in the Evaluation Process
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Tanesia Beverly, University of Tennessee at Knoxville, tbeverl2@utk.edu
Abstract: This poster presentation will disclose my experience in conducting an outcome evaluation for a program with new program staff and limited resources as part of my first evaluation project. This presentation will discuss the difficulties I encountered working with new program staff with limited knowledge of program evaluation including gathering background information, creating a logic model, and gathering/collecting data from minority undergraduate students. The presentation will also discuss the lessons learned while conducting the evaluation. Moreover, I will provide suggestions and strategies for novice evaluators on how to engage new program staff and primary stakeholders in the evaluation process and how to follow culturally competent guidelines while working with diverse populations.
Poster 143: How do You Measure That? Findings From a Review of Youth Development Measurement Tools
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Sharika Bhattacharya, JBS International Inc, sbhattacharya@jbsinternational.com
Christy Olenik, JBS International Inc, colenik@jbsinternational.com
Abstract: This presentation will give the results of an investigation into the tools available to measure youth development outcomes. The United States Agency for International Development (USAID) is interested in more strategically documenting the impact of its programs for young people - specifically with regard to life skills, workforce readiness soft skills, and developmental assets. To support this effort, a systematic review of available tools was conducted. This review included collecting measurement tools through outreach to youth serving organizations and researchers, as well as a scan of organization websites and resource repositories. As a result, more than 50 measurement tools were reviewed by the research team covering multiple concept areas. Tools were evaluated based on, among other things, relevancy to the outcomes of interest, adaptability to various cultural contexts, length, and reliability. Ultimately, the list was whittled down to a set of tools recommended for use by USAID Missions around the world.
Poster 167: Process Evaluation of a Teacher Evaluation Initiative
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Cynthia Blitz, Rutgers University, cindy.blitz@gse.rutgers.edu
William Firestone, Rutgers Graduate School of Education, william.firestone@gse.rutgers.edu
Dessi Kirova, Rutgers University, dessi.kirova@gse.rutgers.edu
Abstract: In response to reform demands from educational stakeholders that are backed by federal incentives for change, most states are currently in the process of introducing a new generation of teacher evaluation systems that are intended to make performance measurement and feedback more rigorous and useful. This presentation describes the design and implementation of a process evaluation to assess the piloting of a new teacher evaluation system in a Northeastern state. We provide an overview of some of the key challenges involved in evaluating a statewide implementation of a new teacher evaluation system and how these challenges may be adequately addressed, methodologically, through a combination of qualitative and quantitative methods. In particular, we present evidence that underscores the importance of gauging the way in which different stakeholders (e.g., teachers, administrators) experience the implementation and make sense of the new teacher evaluation system.
Poster 62: Evaluating Preschool Classroom Quality: Fluctuations and Implications for Impacts
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Lisa Boyce, Utah State University, lisa.boyce@usu.edu
Audrey Juhasz, Utah State University, aud.even@aggiemail.usu.edu
Abstract: In these times of budget cuts, it is especially important for pre-K programs to be able to document their impacts with evaluation results. An important part of these evaluation results is the documentation of program quality. However, program quality is not a static component of a pre-K classroom. Program quality can fluctuate throughout the year and be influenced by many factors including professional development activities, mentoring of teachers, knowledge of the children in the classroom, and children's knowledge, comfort level, and behavior in the classroom. A two year evaluation of a Head Start program will be used in this presentation to illustrate the fluctuation of classroom quality and examine 19 trajectories of classroom quality over the two year period. Discussion will focus on stability of classroom quality indicators in relation to evaluations of child outcomes and professional development activities.
Poster 11: The Fine Line Between Evaluation and Program Design: A Capacity Building Workshop for Extension Staff
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Sherry Boyce, University of Minnesota, sboyce@umn.edu
Pamela Larson Nippolt, University of Minnesota, nippolt@umn.edu
Abstract: Roger Rennekamp asserts that "A well designed program will result in significant and lasting changes in people and the conditions in which they live. Consequently, a program is defined as a sequence of intentional actions and events organized in a manner that they result in valued outcomes for a clearly defined audience." This demonstration will showcase how an Extension evaluation team used a program design framework to build the capacity of Extension professionals to plan for and integrate evaluation from the beginning of a program. Using two perspectives on program design, Extension professionals learn design principles that enable them to design from both "outside" the program and the experience "inside" the program. This can lead to a stronger design, program theory, and more effective starting point for evaluation. This way of talking about program design can be applied across many types of programs, content areas, and levels of staff expertise.
Poster 153: HIV/AIDS Linkage to care and adherence in rural settings
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Russell Brewer, Louisiana Public Health Institute, rbrewer@lphi.org
Sarah Chrestman, Louisiana Public Health Institute, schrestman@lphi.org
Terry Estes, Southwest Louisiana AIDS Council, terrye@slac.org
Thomas Huseby, Southwest Louisiana AIDS Council, thuseb@lsuhsc.edu
Angela Hursey, Southwest Louisiana AIDS Council, ahurs1@lsuhsc.edu
Megan Wright, Louisiana Office of Public Health, megan.wright@la.gov
Abstract: The Southwest Louisiana AIDS Council (SLAC) was funded by Elton John AIDS Foundation and ViiV Southern Initiative-Positive Action to break down barriers that prevent individuals living with HIV from receiving HIV care, treatment and support. This poster highlights the findings and lessons learned from the Access to Care and Treatment Adherence interventions conducted by SLAC in rural Louisiana. The Facilitating Access to Coordinated Treatment intervention works closely with the Treatment Adherence Program (TAP). The TAP interventionist meets with clients to develop treatment adherence plans and routinely meets with clients on HAART. Through this support, clients are able to improve their treatment adherence. A combination of clinic, state surveillance, and qualitative interview data is used to evaluate the program, which can be compared to similar linkage to care navigation programs in the state without TAP to determine if there is improvement in retention in medical care over time due to TAP.
Poster 198: Integrated Pest Management for Vegetable: A Program Evaluation to Determine Value
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Gerald Brust, University of Maryland, jbrust@umd.edu
Teresa McCoy, University of Maryland, tmccoy1@umd.edu
Andy Lazur, University of Maryland, lazur@umd.edu
Abstract: The University of Maryland Extension (UME) Integrated Pest Management (IPM) vegetable program works with commercial vegetable producers to enhance production and profitability. A program evaluation effort was undertaken in winter 2012-2013 to assess high-risk pesticide use, the number of high-risk applications per season per farm and usefulness of a Commercial Vegetable Production Recommendations (CVPR) manual. Results indicate that 51% of respondents have reduced their use of high-risk pesticides; that most reduction has been in the range of 2-4 per season; and that practice changes include greater use of reduced-risk pesticides and cultural controls. Over 65% of respondents use the production recommendations publication 5 or more times per season. The overall total value in dollars saved or earned because of knowledge and skill gained is estimated to be $31 or more per acre by 54% of respondents.
Poster 113: Proceed With Caution: A Qualitative Investigation of Caution Indices With Self-report Data
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Leia Cain, University of South Carolina, leiacain@email.sc.edu
Jin Liu, University of South Carolina, liu99@email.sc.edu
Christine DiStefano, University of South Carolina, distefan@mailbox.sc.edu
Fred Greer, University of South Carolina, greerf@mailbox.sc.edu
Cristina Sirbu, University of South Carolina, sirbu@mailbox.sc.edu
Abstract: Self-report surveys are often used to collect data in evaluation studies, but data manipulation, both purposeful and not, can harm the accuracy and trustworthiness necessary for sound evaluation. The interpretation of data resulting from self-report surveys is possibly limited by response bias. In this study, open-ended surveys and personal interviews were utilized in order to examine social validity and possible response bias within a group of respondents who participated in data collection for a social and behavioral screener designed for preschool children. Respondents were randomly chosen and sent open-ended response surveys from the group of respondents who were flagged through the Behavioral and Emotional Screening System due to potential response issues. Of those who replied, 20% were invited to participate in personal interviews. The results help to inform researchers of the complexities underlying self-report data and its relationship to the obtained data.
Poster 191: The Times, They Are a-Changing: A Historical Look at Evaluation Approaches Used in Standard Setting
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Jade Caines, University of Pennsylvania, jade.caines@gmail.com
Abstract: Although standard-setters regularly follow universal guidelines and procedures when conducting standard-setting studies, it is still a highly evaluative and policy-driven process that relies on human judgment. Formally defined as the specification of a minimally acceptable level of performance on a task, standard setting is the process of determining passing scores, known as cut scores, on an assessment (Cizek, 2012; Cizek & Bunch, 2007; Crocker & Algina, 1986; Glass, 1978). This paper describes three evaluative approaches historically used in standard setting research: Evaluating elements of the process, evaluating panelist judgments, and evaluating context. Also, this paper will thoroughly examine the types of evaluation research questions asked over time. Finally, ethical considerations and policy relevance will also be discussed.
Poster 45: A Framework for the Evaluation of Research Projects
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Patricia Campbell, Campbell-Kibler Associates Inc, campbell@campbell-kibler.com
Emorica Hill, Harvard University, emorcia_hill@hms.harvard.edu
Abstract: While there are many useful tools to assist in program evaluation there are few tools and little guidance for the evaluation of research projects. The evaluation of research has traditionally been done through peer review and citation indices rather than by evaluators who engage in structured interpretation using specific evaluation-related tools. As more funders require research proposals have evaluation plans, evaluators are becoming involved in the evaluation of research projects. In this paper, Campbell and Hill, both evaluators and researchers, will present their Framework for the Evaluation of Research Projects. Our work focuses on four areas: proposal review and award, study implementation, management and administration and dissemination and translation. The paper describes assumptions, benchmarks and possible dashboards for each area. It also provides distinctions between frameworks for the evaluation of programmatic interventions and research projects.
Poster 221: Linking the School, Family, and Clinic: Improving Continuity of Care for Children With Asthma
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Laurie Carpenter, University of Michigan, lauriemc@umich.edu
Maggie Wilkin, University of Michigan, mwilkin@umich.edu
Laurie Lachance, University of Michigan, lauriel@umich.edu
Noreen Clark, University of Michigan, nmclark@umich.edu
Abstract: Improving health outcomes for children with asthma requires a comprehensive and coordinated approach, an approach which enhances continuity of care by connecting the many environments in which children live and learn. Over the past five years, fourteen Missouri programs funded by the Missouri Foundation for Health have implemented a variety of interventions which have connected families, schools, and health care providers and that have resulted in enhanced continuity of care for children with asthma in their communities and improved asthma-related health outcomes. Statistically significant program evaluation findings reported by parents of children in the program include fewer asthma symptom days and nighttime awakenings, less rescue inhaler use and asthma-related health care use, and less absenteeism from school due to asthma. This presentation will illustrate the relationship between outcomes and increased linkages across environments, identifying program components correlated with improved outcomes.
Poster 234: Building a Framework for a Contextual Model of a Multi-site Project
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Aundrea Carter, University of North Carolina at Greensboro, adcarte2@uncg.edu
Christine Meyer, University of North Carolina at Greensboro, ccmeyer@uncg.edu
Holly Downs, University of North Carolina at Greensboro, hadowns@uncg.edu
Lindsey Varner, University of North Carolina at Greensboro, l_dunn@uncg.edu
Abstract: The current study will explore frameworks of context as a necessary condition for planning and evaluating multi-site projects. Using those frameworks in tandem, a contextual model will be developed for a multi-site project that enables considerations of partnerships when planning and evaluating these programs. The authors will provide an example for visually modeling the complexity and multidimensional aspects of a program's context. Context shapes not only the program development but also what the evaluation attends to. Thus, attention will be given to the influence of this context on evaluation activities such as planning, data collection, analysis, reporting from the perspective of a multi-site Common Core mathematics professional development program.
Poster 186: Development of a School-Level Tool to Monitor Implementation of a Multi-Tier System of Supports
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Jose Castillo, University of South Florida, jmcastil@usf.edu
Karen Childs, University of South Florida, kchilds2@usf.edu
Kevin Stockslager, University of South Florida, kstocksl@usf.edu
Abstract: Data on the use of a multi-tiered system of supports (MTSS) to meet the academic and behavioral needs of students suggests that many school districts are implementing the model. However, estimates suggest that most districts do not have a process for evaluating their implementation. One hypothesis for this finding is a lack of access to instrumentation that can reliably and validly inform implementation at the school-level. Therefore, the current study reports on the development of an instrument to monitor implementation of an MTSS at the school-level. The process used to develop and refine items, gather expert and user feedback, create an initial version of the tool, and collect data to examine reliability and validity will be described. The information used to inform each step of the process also will be provided. Finally, implications for future research on the tool and potential uses for informing MTSS implementation will be discussed.
Poster 81: Evaluating the Social Validity of Psychosocial Support Interventions for Children Orphaned by HIV/AIDS in Zimbabwe
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
George Chitiyo, Tennessee Technological University, gchitiyo@tntech.edu
Morgan Chitiyo, Duquesne University, chitiyom@duq.edu
Darlington Changara, Midlands Aids Caring Organization, director@maco.co.zw
Abstract: The AIDS epidemic has created many orphans around the globe. A majority of these orphans live in sub-Saharan Africa. Because of poverty, the stigma associated with HIV/AIDS and stress, due to the loss of their parent(s), children orphaned by HIV/AIDS face daunting challenges in their struggle to cope with these realities. Different psychosocial and educational interventions have been developed to help the children to overcome these challenges. However, few of the interventions have been empirically tested to determine their efficacy. This presentation is based on a study which sought to evaluate the social validity of a psychosocial support program using a group of teachers who had received training and were participating in the implementation of the program across their schools. The results indicated high acceptability ratings of the program.
Poster 177: Evaluation Practice at a Global Health Non-profit Organization
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Korinne Chiu, Physicians for Peace, kchiu@physiciansforpeace.org
Natalie Brogan, Physicians for Peace, nbrogan@physiciansforpeace.org
Abstract: This presentation describes the history and evaluation of a teen parent mentoring program created in the United States and adopted in the Dominican Republic through a global health non-profit organization. Program and evaluation staff from the non-profit organization will discuss how the need for the program was identified, initial program implementation, and the evolution and contextual changes of program delivery due to evaluation. Measures used to evaluate the program will be presented and lessons learned from evaluation findings will be discussed. Collaboration between local partners in the Dominican Republic in order to assess needs, adapt program areas, create valid measures, and continue in-country monitoring and evaluation efforts will be emphasized as key components to ensure program and evaluation practice success. Practices used within the organization for program delivery and evaluation will be explained.
Poster 2: Understanding Evaluation From the Inside Out: The Internal/External Evaluator Partnership
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Korinne Chiu, Physicians for Peace, kchiu@physiciansforpeace.org
Stacy Johnson, The Improve Group, stacyj@theimprovegroup.com
Abstract: This presentation describes an internal-external evaluation partnership to introduce and develop an evaluation for a global health non-profit organization whose core purpose is to train, support, and empower healthcare professionals working with the world's underserved populations. An external evaluation firm was contracted to design and implement an evaluation assessing the effectiveness of the organization's programs and training model by conducting site visits, reviewing data, and surveying and interviewing stakeholders. Simultaneously, an internal evaluator was hired to build capacity and organizational support for evaluation, as well as collaborate with the external evaluation firm. This internal-external evaluation partnership established a framework for evaluation and garnered support from stakeholders, demonstrating the use and importance of evaluation findings and recommendations. Benefits, challenges, and qualities important for internal-external collaboration as well as recommendations implemented within the organization will be discussed.
Poster 179: Multi Interventions: Evaluation of HIV/AIDS Interventions to iMprove Linkage to Care
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Sarah Chrestman, Louisiana Public Health Institute, schrestman@lphi.org
Snigdha Mukherjee, Louisiana Public Health Institute, smukherjee@lphi.org
Russell Brewer, Louisiana Public Health Institute, rbrewer@lphi.org
Peter Gamache, Louisiana Public Health Institute, pgamache@lphi.org
Thomas Carton, Louisiana Public Health Institute, tcarton@lphi.org
Megan Wright, Louisiana Office of Public Health, megan.wright@la.gov
DeAnn Gruber, Louisiana Office of Public Health, deann.gruber@la.gov
Abstract: Louisiana is one of five Positive Charge sites that were funded by AIDS United and Bristol Myers Squibb in 2010 to enhance access to HIV care. Louisiana's PC initiative, led and evaluated by the Louisiana Public Health Institute in partnership with ten other partners is focused on linking newly HIV diagnosed individuals and those living with HIV infection who are not in care or have fallen out of care into primary medical care through four interventions: patient/peer health navigation, pre/post-release case management, disease intervention specialists, and linkage case management. Overall as of June 30, 2012, 682 clients were enrolled and 493 were linked to medical care within 6 months for a linkage rate of 73%. A combination of state surveillance and clinical data is utilized to compare the effectiveness of the interventions on linkage and retention as well as evaluating differences between rural and urban navigation sites.
Poster 148: Lessons Learned From Conducting Outcome Evaluations for Community-Based Programs Among Youth
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Sarah Chrestman, Louisiana Public Health Institute, schrestman@lphi.org
Snigdha Mukherjee, Louisiana Public Health Institute, smukherjee@lphi.rog
Shamsi Soltani, Louisiana Public Health Institute, ssoltani@lphi.org
Kaitlyn Marchesano, Louisiana Public Health Institute, kaitlyn.marchesano@lphi.org
DeAnn Gruber, Louisiana Public Health Institute, deann.gruber@la.gov
Abstract: The Louisiana Office of Public Health - STD/HIV Program is implementing the evidenced based intervention (EBI) SiHLE across Louisiana in community settings as part of the Teen Pregnancy Prevention Program funded by U.S. Department of Health and Human Services. The goal is to decrease risky sexual behaviors, promote gender and ethnic pride, and increase HIV knowledge. In addition to targeting youth, facilitators are African American females ages 18 to 21 or "near peers". To date 400 girls have participated. Data from years 1 and 2 is being analyzed to assess whether the intervention administered by community based organizations (CBOs) yields similar results to the original SiHLE, which was conducted in schools. The presentation will discuss challenges and strategies used to implementing an EBI with CBOs who have limited EBI and evaluation experience, maintaining fidelity, and collecting longitudinal data on sensitive sexual risk behaviors. Outcome data and lessons learned will be presented.
Poster 140: Evaluating the Impact of Conferences: Benefits of Employing a Mixed-Methods Approach
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Christina Chung, Centers for Disease Control and Prevention, vym4@cdc.gov
Abstract: Conferences bring together a network of professionals in an effort to exchange and acquire new information. What is the value of this exchange and how do we assess its impact on participants? A typical conference evaluation consists of a self-administered survey. In order to gain a more comprehensive understanding of our participants' experiences, we employed a sequential exploratory design utilizing both qualitative and quantitative methods to evaluate an international public health conference attended by over 1,600 guests. Over the course of the three and a half-day conference, 241 intercept interviews were conducted with participants during breaks and poster sessions. Questions varied along the progression of the conference. Qualitative findings informed the development of a follow-up web-based questionnaire. This poster describes some of the findings of this evaluation, while highlighting the value and practicality of employing a mixed methods and real-time approach to understand the value of a conference.
Poster 250: Understanding Evaluation in After-School Programs
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Katherine Ciccarelli, University of North Carolina at Greensboro, kaciccar@uncg.edu
Elise Eifert, University of North Carolina at Greensboro, ekeifert@uncg.edu
Christine Meyer, University of North Carolina at Greensboro, ccmeyer@uncg.edu
Krisha Parker, University of North Carolina at Greensboro, ktparker@uncg.edu
Abstract: The number of students served in after-school programs has increased substantially in recent decades and the impact of after-school programs on youth development is significant. A qualitative study using survey and interview data is being conducted on five after school programs in order to understand their experiences with evaluation, specifically how they conduct evaluation as well as the difficulties that they face conducting evaluation. Based on the literature, we expect to find that common reasons information is not collected are a lack of finances, lack of time, and a lack of understanding on how to conduct evaluations. We also expect to find that common reasons information was not used to guide decision making includes a lack of trust in the information gathered, a lack of understanding, and a lack of time to make the changes the data suggested. We conclude with recommendations for policy and additional research.
Poster 159: Developing Multiple Data Collection Instruments to Capture Rigor, a Hard-to-Measure Phenomenon in Classroom Instruction
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Elizabeth Cooper-Martin, Montgomery County Public Schools, elizabeth_cooper-martin@mcpsmd.org
Helen Wang, Montgomery County Public Schools, helen_wang@mcpsmd.org
Abstract: This evaluation focused on the extent of implementation of middle school, advanced-level courses. The courses were designed to be substantively more challenging than on-level instruction with respect to content, differentiated instruction, and learning products. The challenge faced by the evaluators was how to measure the rigor expected for advanced courses. The evaluators will present the dimensions identified as key for rigorous instruction (i.e., focus on big ideas, critical thinking skills, and student-centered instruction) as well as operational measures used in classroom observations, student surveys, and teacher surveys. Along with dimensions that were common across content areas, the authors identified content-specific dimensions and related measures for three subject areas: English, science, and world studies. The presentation will describe the processes used to establish the dimensions and measurements for a hard-to-measure phenomenon and the approaches used to ensure validity and reliability.
Poster 207: Evaluation as a Tool for Professional Development
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Liesel Copeland, Independent Consultant, hlcopeland@gmail.com
Michael Nestor, The New York Stem Cell Foundation, mnestor@nyscf.org
Martha Soto, University of Medicine & Dentistry of New Jersey, sotomc@umdnj.edu
Krishna Tobin, Ruters University, tobonkr@rwjms.rutgers.edu
Abstract: This presentation will present two case studies of implementing small scale evaluations in a participatory fashion to increase professional development of teachers. To encourage scientist fellows continued reflection of teaching methods and adoption of new methods, they were given the option of implementing evaluation or small scale research projects within their lectures. The use of evaluation would theoretically allow them to construct their own evidence and systematically approach adjustments to their teaching styles. Lessons learned in implementing small scale evaluations with a participatory lens as well as how the evaluation merged with the professional development around teaching will be shared.
Poster 257: Approaches to Assessing the Impact of Biomedical Research: Examples From the National Cancer Institute (NCI)
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
James Corrigan, National Institutes of Health, corrigan@mail.nih.gov
Elizabeth Hsu, National Institutes of Health, hsuel@mail.nih.gov
Joshua Schnell, Thomson Reuters, joshua.schnell@thomsonreuters.com
Duane Williams, Thomson Reuters, duane.williams@thomsonreuters.com
Brian Zuckerman, Science and Technology Policy Institute, bzuckerm@ida.org
Abstract: The ultimate goal of biomedical research is improved health; progress can be observed in various health statistics (e.g., incidence, mortality, and survival rates). Given lags from basic discovery to translation into interventions, and from application of new interventions to changes in health statistics, and the lack of a single definitive approach to assessing biomedical research impact, there is interest in multiple converging approaches to gauging impact. Research funding can be linked to key advances in multiple ways. Challenges faced in assessing impact include both attribution to and strength or directness of linkage to a particular funding source. Methods described have assessed the NCI role in: 1) expert-identified basic and clinical scientific advances; 2) biomarkers; 3) FDA-approved drugs; and, 4) guidelines for cancer treatment, screening & detection, and supportive care. With multiple sources of support common for research efforts, defining the impact of funding from any single source is complex.
Poster 50: Evaluation of a Multi-Site, Out-of-School-Time Program: Contextual, Individual, and Combined Influences on Outcomes
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Amy Corron, United Way of Greater Houston, acorron@unitedwayhouston.org
Roger Durand, University of Houston at Clear Lake, durand4321@gmail.com
Emily Gesing, United Way of Greater Houston, egesing@unitedwayhouston.org
Julie Johnson, Communities-in-Schools of Houston, jjohnson@cis-houston.org
Kevin Kebede, Alief YMCA, kevink@ymcahouston.org
Jennifer Key, Alief Independent School District, jennifer.key@aliefisd.net
Linda Lykos, YMCA of Greater Houston, lindal@ymcahouston.org
Cheryl McCallum, Children's Museum of Houston, cdm@cmhouston.org
Abstract: In this paper the results of an evaluation of a multi-site, out-of-school-time program will be presented. A particular focus will be on evaluation findings concerning the separate and combined impacts of site (or contextual) characteristics and individual participant differences in determining program outcomes. In the aftermath of Hurricanes Katrina and Rita, an out-of-school-time, collaborative program, known as "Houston's Kids," was implemented with the intention of addressing the needs of displaced and other at-risk youth in a single community. An outcomes evaluation was conducted to assess the program's effects on developmental assets as well as on academic achievement among the approximately 300 individual kindergarten through intermediate school participants. True panels of data that tracked changes in the same individual participants over a school year were employed as were data on a sample of control or "comparison" subjects. Hierarchical and multiplicative statistical models were used to analyze the data.
Poster 176: How a Community Collaboration is Creating Comprehensive and Cool Assessments: 21st Century Tools to Support 21st Century Youth
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Laurel Crown, Chicago Public Schools, lbcrown@cps.edu
Wendy Fine, Youth Guidance, wfine@youth-guidance.org
Leslie Beller, Chicago Public Schools, lbeller@cps.edu
Abstract: The "Means and Measures of Human Achievement Toolkit" represents an innovative approach to support the development of 21st century skills that are critical for youths' college, career, and life success. The toolkit, which includes assessment, professional development, and instructional tools, fills a critical gap in the educational and youth development arenas where there is a dearth of reliable, cross-culturally relevant, and developmentally-focused means for promoting and assessing social, emotional, and cognitive competencies. The diverse collaborative of researchers, practitioners, and youth who developed the toolkit participated in an eclectic array of activities, including collaborative factor analysis, cultural linguistic concept analysis, and youth-driven prototyping. The resulting products, including a well-designed formative assessment tool (Human Achievement Quotient), are inspiring a movement of K-12 teachers, human service providers, workforce programs, employers, colleges, parents, and even youth to chronicle and promote 21st century skill development across the lifelong learning spectrum.
Poster 173: Evaluability assessment of Brazilian Ministry of Health's strategic project "Training and Quality Improvement of Health Care Network" (QualiSUS-rede)
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Marly Cruz, Fiocruz Foundation, marly@ensp.fiocruz.br
Gisela Cardoso, National School of Public Health, gisela.cardoso@gmail.com
Egléubia Oliveira, Fiocruz Foundation, biaol@ensp.fiocruz.br
Ana Gurgel, Brazil Ministry of Health, analucia.gurgel@saude.gov.br
Margarete Oliveira, Brazil Ministry of Health, margarete.oliveira@saude.gov.br
Aline Duque, Fiocruz Foundation, alineduque13@yahoo.com.br
Glaydes Alves, Fiocruz Foundation, glaydesalves@yahoo.com.br
Carlos Leonardo Cunha, Fiocruz Foundation, leocunhama@gmail.com
Rafaela Souza, Fiocruz Foundation, rafaelacbs@globo.com
Abstract: Evaluability assessment of Brazilian Ministry of Health's strategic project "Training and Quality Improvement of Health Care Network" (QualiSUS-rede) was conducted, between 2012-2013. This cooperation project among the World Bank and Brazilian Ministry of Health aims to contribute to the qualification of care, health management and developing technologies, through the organization of thematic and regional networks of health care in 15 regions of Brazil. The evaluability study involved the following activities: meetings planning with project managers and external consultants, stakeholders' workshops, review of goals and indicators, development of data collection instruments, construction of 15 logical models, and the elaboration of external evaluation parameters. The evaluability assessment contributed to increase knowledge about the intervention strategies in the 15 regions involved, and the steps to be followed in the evaluation itself. Important to emphasize the formative character of this initial stage, providing revisions and adjustments for this intervention.
Poster 16: Evaluation of Health Systems in Brazil: A Collaborative Evaluation Experience
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Marly Cruz, Fiocruz Foundation, marly@ensp.fiocruz.br
Paulo Tarso, DEMAS/MS, pttarso@gmail.com
Elizabeth Moreira dos Santos, National School of Public Health, betuca51@gmail.com
Abstract: This presentation cover current initiatives in M&E undertaken by the MoH-Brazil aiming to describe and to identify achievements and challenges to the effort of integration and coordination of the different M&E initiatives of the Minister. The presentation comprises a short description of the initiatives looking at the Sala de Situa+ºão para Gestão Estratégica (Managment Observatory) as an example of integrating technology and expertise to provide on timely response for reflexive management. The observatory integrates 250 indicators, some of them such as those related to emergence care that are followed on daily basis aiming to supply federal government with strategic information and possibility to mobilize expertise for problem solving.
Poster 117: Learning and Teaching Collaborative Program Evaluation: Lessons From a Community Based Learning Project With Graduate Students in Conflict Resolution
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Rachel Cunliffe, Portland State University, rachel.cunliffe@pdx.edu
Rachel Foxhoven, Insight Development Group, rachel.foxhoven@gmail.com
Kelsey Ferrell Ferrell, Portland State University, kelseyferrell@gmail.com
Chantal Downing, Portland State University, cdowning@pdx.edu
Abstract: Conflict Resolution program evaluation is notoriously sparse. Its comparative rarity is due to the complexity of operationalizing many of the desired outcomes. Recent evaluation of conflict resolution graduate programs suggests that evaluation should be taught. A graduate program in conflict and resolution in the northwest chose collaborative program evaluation as the logical and most appropriate approach to evaluation in the conflict resolution and peace-building arenas. Four community organizations all offering restorative justice services agreed to participate in what was known as the collaborative program evaluation learning community for the ten week quarter. Graduate students were introduced to evaluation techniques in the course of their participation on evaluation teams. Community organizations and student attitudes and experiences were evaluated and lessons learned will be shared.Of particular interest is the challenge of facilitating the development of skills in and attitudes conducive to collaboration.
Poster 139: Using the Demographic and Health Survey (DHS) Calendar to Estimate the Effect of Family Planning Use on Birth Weight
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Alexis D'Agostino, John Snow Inc, alexis_dagostino@jsi.com
Amanda Pomeroy, Strengthening Partnerships, Results, and Innovations in Nutrition Globally Project (SPRING), amanda_pomeroy@jsi.com
Abstract: Using a novel approach to analyzing Demographic and Health Survey (DHS) reproductive calendar data, the USAID-funded SPRING project estimated the relationship between family planning (FP) and birth outcomes. We constructed variables on FP use and length of interpregnancy interval (IPI) for women who had at least two births and whose penultimate birth was at least 6 months from the start of the calendar for the India 2005-06 DHS (>100,000 women). We ran a linear regression of the effect of family planning use on IPI length, controlling for mother's background, household characteristics, and symptoms during pregnancy. A separate regression estimated the effect of IPI length on birth weight using the same set of controls. Long-term contraceptive use increased IPI length by 4.6 months (SD=0.435) and short-term use increased IPI by 4.2 months (SD=0.265). Compared to IPIs of less than six months, IPIs above six months increase birth weight by 98.1g (SD=48.3).
Poster 241: Long-term Effects of Participation in Full-Service Partnerships on Psychiatric Inpatient Unit and Crisis Services Utilization
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Rajima Danish, Ventura County Behavioral Health, rajima.danish@ventura.org
Abstract: Full Service Partnership (FSP) programs offered by a California county behavioral health agency to children, transitional aged youth, adults, and older adults are evaluated for their short- and long-term impact on consumer utilization of other behavioral health services. FSPs provide intensive services with a whatever it takes approach to under- and unserved populations. Utilization of inpatient psychiatric hospital and crisis services prior to enrollment in the FSP, during the FSP episode, and after discharge from the FSP are analyzed. Outcomes are compared to a control group of consumers receiving typical outpatient mental health services to determine if differences in service utilization exist. Possible interactions between common sociodemograpahic factors such as gender, ethnicity, age, and insurance status and service utilization are also investigated. Preliminary analysis demonstrates promising outcomes for consumers enrolled in an FSP. Sharing of these evaluatio n procedures and findings lend support and ideas to other behavioral health evaluators.
Poster 187: Challenges of a Multi-Site Evaluation of a Science, Technology, Engineering, and Mathematics (STEM) Research Program for Teachers
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Tiffany Davis, Georgia Institute of Technology, tiffany.davis@ceismc.gatech.edu
Meltem Alemdar, Georgia Institute of Technology, meltem.alemdar@ceismc.gatech.edu
Abstract: The National Nanotechnology Infrastructure Network (NNIN) Research Experience for Teachers (RET) is a multi-site program designed to expose high school and community college science teachers to Science, Technology, Engineering, and mathematics (STEM) fields and Nanoscale Science Engineering (NSE). Through a summer research experience and ongoing follow-up support, teachers are required to develop learning units for integrating scientific concepts into the classroom. This poster describes the ongoing, mixed-methods evaluation of four sites participating in the NNIN RET. Focusing on formative and summative evaluation throughout the 3-year program, the evaluation employs both quantitative and qualitative research methods, including surveys, focus groups and document review. In this section, we will discuss the challenges faced in the evaluation of a multi-site program with a small sample size and present some suggested strategies.
Poster 39: A Collaborative, Multiple Design, Process and Outcomes Evaluation of a Local Habitat for Humanity Project
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Phillip J Decker, University of Houston at Clear Lake, decker@uhcl.edu
Roger Durand, University of Houston at Clear Lake, durand4321@gmail.com
Christine Eriks, University of Houston at Clear Lake, cmeriks@gmail.com
Natalie Ainsworth, Bay Area Habitat for Humanity, nainsworth@bahfh.org
Rachel Ward, Bay Area Habitat for Humanity, rward@bahfh.org
Abstract: In this poster presentation the findings from a collaborative, multiple design, process and outcomes evaluation of a Habitat for Humanity (HFH) project will be reported. Background: Habitat for Humanity is a widely known, charitable housing organization that builds affordable housing in partnership with families in need. Evaluation questions: What problems were experienced by partner family adult members and their resident children with an HFH project's implementation? What were the project's social and financial outcomes? Data and Methods: Utilizing multiple, varying time series, process and outcomes evaluation designs, baseline evidence was gathered from participants in a local, Houston area HFH project, including members of the local board of directors, volunteers, faith-based stakeholders, construction workers, partner family adults and their children. The evaluation was designed collaboratively with project participants working with professional evaluators. Follow-up data are now being gathered. Discussion: Both the collaboration and evaluation results will be discussed.
Poster 37: The Predictive Power of Self-Evaluations in Assessing Educational and Academic Outcomes
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Phillip J Decker, University of Houston at Clear Lake, decker@uhcl.edu
Roger Durand, University of Houston at Clear Lake, durand4321@gmail.com
Lee Revere, University of Houston at Clear Lake, revere@uhcl.edu
Abstract: In this poster the results of research on the power of self-evaluations of competencies to predict subsequent actual and perceived academic outcomes, especially job progression, will be reported. Background: Academic accrediting organizations are now requiring educational institutions to report student competency and alumni job outcomes. Self-evaluations are a tool increasingly being utilized to meet this requirement. The evaluation problem: Do self-evaluations predict well subsequent outcomes, including alumni job progression? Data and Methods:Survey evidence about job progression was gathered from 150 former students enrolled in a single graduate program and matched to their archived self-assessments. Analysis: Linear and non-linear predictive, statistical models with numerous controls for potentially extraneous variables were utilized in data analysis. Findings:The findings establish the predictive power of self-evaluations.
Poster 54: Place Based Needs Assessment
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Ann Del Vecchio, Alpha Assessment Associates LLC, ann.delvecchio@gmail.com
Abstract: Needs Assessments are conducted routinely as part of the strategic planning process, as a section of many grant proposals, and often to identify the specific needs for professional development, training, or capacity development before a plan is implemented. How is the context and culture of the place wherein the needs assessment is conducted a factor in: 1) the methods and sources used to collect data; 2) the type of data analyses used; 3) the development and format of the needs assessment; 4) how the needs assessment is used; and 5) how the needs assessment can be used a s the foundation for program / project evaluations. This poster will provide examples of two place based needs assessments that were used to secure federal funding and that formed the foundation of the evaluations of the funded projects. It will demonstrate examples of methods, analyses and summary formats linked to place.
Poster 166: Multi-Site Evaluation of Spiritual Care in Southern California Hospitals
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Adrienne Dellinger, Vital Research LLC, adellinger@vitalresearch.com
Gwen Uman, Vital Research LLC, guman@vitalresearch.com
Harold Urman, Vital Research LLC, hurman@vitalresearch.com
Abstract: A foundation-funded Spiritual Care Demonstration Project focused on improvement of the quality of spiritual care provided by palliative care teams at nine Southern California hospitals over a two year period. The overarching goal of the project was to examine how and to what extent each hospital fostered, facilitated, and sustained the development of quality spiritual care. This multi-site evaluation included both quantitative and qualitative measures, with data collected through structured interviews of patients and families, staff interviews, site visits, document analysis, and longitudinal data reported by each site. Evaluating diverse hospitals with varied data collection capabilities and different database systems created a unique situation for comparative analysis. Results helped to elucidate the strengths and challenges in working with complex hospital systems, and provided insight into strategies for managing large multi-site projects.
Poster 119: Family Support and Natural Disasters: The Impact on Adolescent Substance Use
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Susan Depue, Missouri Institute of Mental Health, susan.depue@mimh.edu
Abstract: This presentation will examine data on 4,414 youth who were involved in natural disasters. The focus is the impact of the natural disaster, as influenced by family support, on substance use. Analysis shows a significant interaction between the two independent variables when looking at alcohol and marijuana use. The interaction suggest that the most dramatic difference in a natural disaster's impact on past month use occurs in those adolescents who are either in the top or bottom quartiles for family support. There were also significant main effects on substance use for both being involved in a natural disaster and family bonding. These results suggest that evaluators should consider the effect family bonding has on substance use for those who have been through a natural disaster.
Poster 76: Bridging the Diversity Among Local Evaluations: Cross-Site Findings of 24 Child Welfare Discretionary Grants
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Jennifer Dewey, James Bell Associates Inc, dewey@jbassoc.com
Grace Tipton, James Bell Associates Inc, tipton@jbassoc.com
Carolyn Sullins, James Bell Associates Inc, sullins@jbassoc.com
Joanna DeWolfe, James Bell Associates Inc, dewolfe@jbassoc.com
Chi Connie Vu, James Bell Associates Inc, vu@jbassoc.com
Abstract: This paper presents process and outcome evaluation methods and results from the cross-site evaluation of 24 Family Connection Discretionary Grants. Federally-funded in 2009, grants supported demonstration projects to reconnect family members with children who were in or at risk of entering foster care. Authors leveraged local evaluation designs, data collection plans, and anticipated outcomes to develop a cross-site evaluation that assessed findings specific to four types of programs (kinship navigator, family-finding, family group decision-making, residential family treatment) and grantees as a whole. Authors used grantee applications, plans and other documents, standardized local evaluation reports, and interview data to document target populations, service models and key activities, child and family-level outcomes, and organizational and system-level outcomes. Conclusions and recommendations to funders and the child welfare field to increase program impact are provided. Authors also document multiple key improvements to the cross-site design implemented with grantees funded in 2011 and 2012.
Poster 5: Examining the Utility of a Goals-based Evaluation of a Cross Cutting Public Health Program in a Post-disaster Setting
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Nicholas Di Meo, Centers for Disease Control and Prevention, ndimeo@cdc.gov
Angela Wood, Centers for Disease Control and Prevention, awood@cdc.gov
Abstract: In response to the devastating earthquake that hit Haiti in January of 2010, Congress provided funding to the Centers for Disease Control and Prevention's (CDC) Health Systems Reconstruction Office (HSRO) to aid the reconstruction of public health systems in Haiti. The earthquake killed more than 200,000 people, leveled basic infrastructure, and gave rise to a deadly cholera epidemic. CDC's response was rapid and included both emergency response and long-term public health infrastructure and capacity development. This session will describe how HSRO used a goals-based evaluation framework to mobilize partner support and resources, guide strategic planning, and track implementation and progress toward outcomes. This session will also describe an integrated strategic planning and evaluation tool used by the evaluators and examine the strengths and weaknesses of this approach and share lessons learned about the use of this model in a post-disaster setting.
Poster 44: Looking Long-Term: How to Utilize a Decade of Applied Epidemiology Fellowship Data to Evaluate Effectiveness
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Virginia Dick, Council of State and Territorial Epidemiologists, vdick@cste.org
Amanda Masters, Council of State and Territorial Epidemiologists, amasters@cste.org
Abstract: For almost 10 years, the Centers for Disease Control (CDC)/Council of State and Territorial Epidemiologists (CSTE) Applied Epidemiology Fellowship has provided recent masters and doctoral graduates training, mentorship, and field experience in state and local public health agencies. In 2012, CSTE began conducting a thorough longitudinal evaluation to determine the long term impacts of the program on the fellows who have participated over the past 9 years, the mentors who provided guidance and expertise, and the state and local agencies who served as host agencies for fellows. This presentation will review the findings from this extensive evaluation. This contributes to the field of evaluation in public health and education by examining the potential long term outcomes associated with field based training opportunities.
Poster 256: The Combination of Formal Theory and Stakeholder Theory: From a Literacy Intervention Program Evaluation
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Shufang Dong, University of Rochester, shufangd@rochester.rr.com
Abstract: 21st century requires effectiveness more than ever. This proposal is to show how effective a literacy intervention program is in multiple aspects. Literacy intervention program at School Seeds is based on formal theory and stakeholder theory. Chen states that formal theory-based interventions have acquired high prestige, while stakeholder theory-based interventions have been held in low esteem. This study is to examine the two types of interventions via a case study of program evaluation to literacy intervention program at School Seeds. It is noted by investigating the program that a literacy program could be more than improving the literacy skills of school kids. Instead, in this program, the literacy interventions are multifaceted, ranging from the core goal of improving students’ literacy skills to the community enhancement, from educational issues to social betterment. It is concluded that the combination of formal theory and stakeholder theory maximized the effectiveness of this literacy program.
Poster 125: A Retrospective Review of the Pandemic Influenza Competitive Announcement Pilot Projects
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Stephanie Dopson, Centers for Disease Control and Prevention, sld9@cdc.gov
Christa Singleton, Centers for Disease Control and Prevention, zbi9@cdc.gov
Abstract: In 2009, the Centers for Disease Control and Prevention (CDC) Office of Public Health Preparedness and Response (OPHPR) Division of State and Local Readiness (DSLR) funded, through a competitive announcement, 55 demonstration projects in 29 state and local public health departments to pilot innovative approaches for influenza pandemic preparedness. Topic areas included Public Engagement, Electronic Laboratory Data Exchange, Countermeasure and State Immunization Systems Integration, Electronic Death Reporting, Collaborative Planning for Essential Healthcare Services, Vulnerable Populations, and Distribution of Antiviral Drugs. The aim of this session is to describe the retrospective review process used for these projects that was conducted in 2012. All grantee applications were reviewed, and interviews were performed to gain a better understanding of projects identified across the seven topic areas that produced results which could be considered promising practices and replicable for preparedness efforts at the state and local level.
Poster 80: Informing Future Practices: Evaluating the Use and Dissemination of an Innovative Health Education Tool in Pediatric Settings
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Jessica Drennan, Centers for Disease Control and Prevention, jdrennan@cdc.gov
Rachel Gaddes, Westat, rachelgaddes@westat.com
Denise Levis, Centers for Disease Control and Prevention, igc1@cdc.gov
Erika Bonilla, Westat, erikabonilla@westat.com
Erika Reed-Gross, Westat, erikareed-gross@westat.com
Katie Green, Centers for Disease Control and Prevention, kpk9@cdc.gov
Rebecca Wolf, Centers for Disease Control and Prevention, rwolf@cdc.gov
Abstract: Using a children's book format, Centers for Disease Control and Prevention (CDC) and partners developed an innovative health education tool, "Amazing Me: It's Busy Being 3!" to increase awareness of developmental milestones among parents of 3-year-olds and engage them in the monitoring of their child's development. CDC partnered with Reckitt Benckiser (makers of Lysol™) and Reach Out and Read (ROR), an organization that promotes early literacy among low-income families, to disseminate "Amazing Me" to parents of 3-year-olds through 250 pediatric clinics across the United States. We surveyed the 250 clinics, evaluating clinics' experience distributing the book and their willingness to continue to reach parents through similar efforts. The evaluation findings include facilitators and barriers to distribution, best practices for distribution, provider opinions on the book, provider engagement of the book with families, and potential for future dissemination of similar health education tools.
Poster 7: Local Capacity Development: Evaluating Performance of Local Partners
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Rachel DuBois, Pact, rdubois@pactworld.org
Olga Yakimakho, Pact, oyakimakho@pactworld.org
Abstract: Faced with expectations of increased accountability for the results of capacity development initiatives, practitioners are striving to move beyond anecdotal stories and output indicators, and towards clearly articulated theories of change that support outcome level measurement of capacity development interventions. In 2011, Pact developed the Organizational Performance Index (OPI) to evaluate the extent to which capacity development investments lead to improvements in partner performance. Pact successfully tested the inter-observer reliability of the OPI using the Kappa Statistic with support from the Rockefeller Foundation. In 2012 Pact collected data from 328 partners in 17 countries, the results of which demonstrate that the length of capacity development support directly correlates with partner performance. In this demonstration, we will show how to evaluate performance using Pact's OPI, showcase the results of research, and compare the OPI with other measurement tools. Attendees will have an opportunity to review OPI data, provide analysis and recommendations.
Poster 236: An iNteractive, Patient-centered Strategy for Health Education Courses Results in Increased Understanding of the Subject Matter, More Engagement, and Adoption of Positive Changes to Their Work Settings
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Vanessa Eaton, American Society of Clinical Oncology, vanessa.eaton@asco.org
Hugo Villar, University of Arizona, 
Doug Pyle, American Society of Clinical Oncology, doug.pyle@asco.org
Abstract: The American Society of Clinical Oncology supports the development of multidisciplinary cancer management practices in low to middle income countries. MCMC courses have been presented 21 times since 2004. They focus on teaching general principles of multidisciplinary cancer management, team-based care, and determining best available treatment options for patients. Courses use common cancer sites to demonstrate these concepts. Prior to 2012, subjects were taught as traditional lectures - one subject per talk. In 2012, an interactive patient-centered strategy was developed that illustrates the stages of treatment by following a patient step-by-step through the continuum of care. The model demonstrates the importance of centering care on the patient, the complexity of delivering quality care, and how the multidisciplinary team works together to achieve optimal outcomes. In order to achieve better interaction, the format included role plays, an audience response system, mock tumor board discussions, and breakout sessions.
Poster 60: Efficacy of Family Skills Training on Preventing Adolescent Substance Use
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Jennifer Ely, University of Pittsburgh, jae39@pitt.edu
Meghan Roman, University of Pittsburgh, mmr67@pitt.edu
Abstract: Literature suggests that parenting practice characterized by monitoring, consistent limit setting, and nurturing communication patterns with children are protective against adolescent substance use and other problem behaviors. Hence, the justification for family based prevention programs. However, low participation rates remain problematic regardless of the incentive for participation. This paper discusses the efficacy of a family skills training implemented with multiple incentives aimed at reducing barriers to participation.
Poster 230: Diversifying Organizations: How to Build Expertise and Capacity to Conduct Evaluation Across Arenas and Methodologies
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Holly Figueroa, Arizona State University, htlewis@asu.edu
Kathryn Hamm, Arizona State University, khamm@asu.edu
Wendy Wolfersteig, Arizona State University, wendy.wolfersteig@asu.edu
Abstract: This presentation will discuss the connections between diversity, evaluation capacity, sustainability, and community impact. There are many questions that evaluators, whether they are independent consultants or part of larger agencies, may ask themselves when new funding opportunities arise: Will I be competitive? Do I need to bring on a methodological or subject matter expert? Am I connected with the right people? This presentation will discuss these questions in terms of diversity from an organizational standpoint - what it means, what types of diversity matters to organizations, and why evaluators should care. We provide case study examples to suggest that an effective blending across multiple categories of diversity (e.g., culture, expertise, networks, etc.), is essential for ensuring the diversity among practitioners and approaches necessary for today's evaluators to be successful across multiple arenas and methodologies. Multi-level diversity also contributes to higher quality evaluation and thus, better informed policy and programmatic decision-making.
Poster 217: Evaluation of the Spontaneous Writing Subtests of the Test of Written Language - Fourth Edition (TOWL-4)
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Erik Fister, University of Kansas, efister@ku.edu
Steven Lee, University of Kansas, swlee@ku.edu
Bruce Frey, University of Kansas, frey@ku.edu
Abstract: The reliable assessment of written expression is an essential component of scholastic evaluation. The purpose of the current study was to establish inter-rater reliability of an augmented scoring rubric for the Spontaneous Writing (SW) section of the Test of Written Language - Fourth Edition (TOWL-4). A sample of 600 8th grade students was recruited from a North American northeastern city and completed the SW section of the TOWL-4. Initial analyses revealed inadequate inter-rater reliability on the basis of the TOWL-4 standardized scoring protocol. An augmented scoring rubric was subsequently developed and evaluated on a sample subset (n= 118). Results showed improvement in inter-rater reliability via consensus estimates of percent agreement. Evaluation-related implications of the current study highlight the need for modifying ambiguously defined writing items on the TOWL-4 and the potential benefit of utilizing a scoring rubric with greater specificity.
Poster 158: How Are Workplace Health Promotion Programs Evaluated and Who is Evaluating Them?
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Carolyn Flanigan, CBF Evaluation, cflanigan@cbfevaluation.com
Abstract: Many businesses who are struggling with rapidly rising employee health care costs have implemented workplace health promotion programs to 1) assess health risks, 2) reduce those risk factors, and 3) promote socially and environmentally healthy lifestyles. Although businesses invest a lot of money in these programs, they often give slight attention to the evaluation component. There is little current research on how workplace health promotion programs are evaluated and who is evaluating them. I propose to interview at least ten experts in the field of workplace health promotion and ask them the research question: How are workplace health promotion programs evaluated and who is evaluating them? To obtain the perspectives of those who are directly involved with these programs, I'll contact businesses who offer the programs and interview or survey program administrators as well as upper management. This study will contribute knowledge to two arenas: workplace health promotion and evaluation.
Poster 239: Research Capacity Building for the Accounting Department's Research Team of Rajamangala university of Technology Lanna Chiangmai: An Application of Empowerment Evaluation
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Noppon Foosang, Sukhothai Thammathirat Open University, liver-149@hotmail.com
Sungworn Ngudgratoke, Sukhothai Thammathirat Open University, sungworn.ngu@stou.ac.th
Nalinee Na Nakorn, Sukhothai Thammathirat Open University, nalinee_na_nakorn@hotmail.com
Abstract: In order to increase research productivity of the department of accounting, we adopted an empowerment evaluation approach (Fetterman & Wandersman, 2005) as a tool for providing strategies for improvement in research productivity of the department of accounting. We considered empowerment evaluation suited our improvement goals because good improvement must start from the inside and empowerment evaluation will offers participants to reflect their weaknesses and strengths and to find their strategies to improve their practices. This study is a small scale empowerment evaluation, specifically there are 7 faculty members participating in the empowerment evaluation. However, the outcomes of evaluation and strategy implementations show strengths of empowerment evaluation.
Poster 192: Conducting a Process Evaluation and Assessing Treatment Fidelity of Kin KeeperSM: A Community-Based Participatory Research Prevention Intervention
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Sabrina Ford, Michigan State University, sabrina.ford@hc.msu.edu
Cristian Meghea, Michigan State University, cristian.meghea@hc.msu.edu
Karen Patricia Williams, Michigan State University, karen.williams@ht.msu.edu
Abstract: Introduction: Evaluation of community-based participatory research (CBPR) is limited, yet funders are increasingly calling for CBPR initiatives and expect evaluations in order to continue funding. This poster will present evaluation methods performed in the first two years of a five year NIH study. The conceptual framework utilizes several models from the CDC, Kellogg Foundation, and Treatment Fidelity Workgroup of NIH. Context: This evaluation is unique for several reasons, the project: 1) is a Randomized Controlled Trial; 2) includes multiple community partners; 3) uses psychometrically standardized instruments tested in a real world setting. Results: The evaluation was performed at baseline and 12 month follow-up and found that utilizing strong conceptual models allows for a comprehensive evaluation of a large project with many components. We also found sound treatment fidelity at 12 month follow-up. Summary: Evaluation procedures for CBPR are feasible and necessary in order to give quality reporting to funding agencies.
Poster 213: The Impact of the Ohio Title IV-E Waiver on County Child Welfare Spending Patterns
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Greg Forehand, Human Services Research Institute, gforehand@hsri.org
Britay Orlebeke, University of Chicago, bochapin@mindspring.com
Abstract: The purpose of the Ohio Title IV-E waiver is to promote public investment in service alternatives to foster care. The historical lack of investment in placement alternatives is partially due to the fact that Title IV-E board and maintenance funds can only be spent on out-of-home care. The flexibility allowed under the waiver should open IV-E funds to a greater variety of uses. Taking advantage of the flexibility to build alternatives to foster care, one would expect that counties participating in the waiver would have a lower utilization of foster care and a concomitant increase in spending for non-placement services relative to counties not participating in the waiver. The methods utilized to collect and analyze county child welfare spending, the changes in spending patterns since the since the start of the waiver in 1998, and the unexpected length of time it took to observe those changes will be presented.
Poster 178: Evaluating Australia's Caring for Our Country Invasive Pest Management Program: A Mixed-Method Analysis of Progress and Opportunities
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Michael Fortunato, Penn State University, heyfortunato@gmail.com
Rama Radhakrishna, Penn State University, brr100@psu.edu
Miriam Verbeek, University of New England, miriam_v@profitfoundation.com.au
Paul Martin, University of New England, pmartin9@une.au
Ted Alter, Penn State University, tra2@psu.edu
Abstract: Caring for our Country is a project aimed at reducing the impact of invasive species and to improve biodiversity and maintain sustainable agricultural practices. The purpose of this evaluative study is to determine the impact of the project on reducing invasive species, identify lessons learnt from on-ground experiences of managing pest animals, and suggest improvements in the design of future NRM program policies and strategies. Data was collected from 130 participants from 40 project initiatives. Interviews used a standard survey protocol, and were administered face-to-face or by telephone. Questions focused on techniques used to reduce the impact of invasive species, effectiveness of techniques used to invasive animal control and effectiveness of communication strategies used. Overall, the findings reveal that the goals of the project, to some extent, have been met. Lessons learned from this effort will be useful as we plan for future program development, implementation and evaluation efforts.
Poster 105: Evaluating the role of peer counselors in "Trauma-Informed Care" for Veterans
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Karen Friend, Pacific Institute for Research and Evaluation, kfriend@pire.org
Jamie Vela, University of Rhode Island, jamie.e.vela@gmail.com
John Stevenson, University of Rhode Island, jsteve@uri.edu
Abstract: In 2008, the Center for Mental Health Services (CMHS), part of the Substance Abuse and Mental Health Services Administration (SAMHSA) launched a new initiative The Jail Diversion and Trauma Recovery Program-Priority to Veterans. Study participants were offered treatment targeting their trauma as an alternative to jail sentences. In Rhode Island, participants included both veterans with trauma, as well as non-veterans who had experience with trauma and met study eligibility criteria. The treatment offered for trauma was TAMAR (Trauma, Addictions, Mental Health and Recovery). In conjunction with TAMAR, peer mentor supports were introduced. This poster examines means for evaluating the role of peer counselors providing trauma-informed care to participants. In this poster, we will provide qualitative and quantitative evidence regarding the roles of the peer counselors, link this to the broader literature on peer support and address some special challenges of incorporating this type of intervention into research by evaluating program effectiveness.
Poster 229: Making the Most of Typical Intellectual Engagement and Student Engagement
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Rafael Garcia, The University of Arizona, ragarci2@email.arizona.edu
Lee Sechrest, The University of Arizona, sechrest@email.arizona.edu
Abstract: Student engagement has come to be regarded as a critical variable in understanding both student and institutional performance in higher education. As yet, however, student engagement seems neither particularly well-defined or measured. A widely known individual measure, Typical Intellectual Engagement, and the institutional assessment National Survey of Student Engagement, appear conceptually similar, but neither has yet been fully explored as measures, nor have they been related to each other. The characteristics of the two measures and their psychometric properties are assessed and compared, and recommendations for future directions of research aimed at understanding and capitalizing on the role of student engagement to improve the effectiveness of higher education, e.g., retention and knowledge transmission, are given in this paper.
Poster 137: Evaluation of a Sober Living Environment Program in an Urban Behavioral Health System
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Cassie Garety, Alliant International University, cgarety@alliant.edu
Yurivia Cervantes, Alliant International University, ycervantes1@alliant.edu
Katherine McCarthy, Alliant International University, kmccarthy@alliant.edu
Abstract: The focus of this evaluation proposal is on the Transitional Housing Program at Treasure Island, which is a Sober Living Environment Program under the Walden House branch of HealthRIGHT 360 in San Francisco, CA. This program is an interim step-down level of care for individuals who have achieved significant progress toward their treatment goals in a residential program, but are not yet ready to fully integrate into community-based living. Individuals may be part of this program for up to three months. The program addresses addiction and related issues, while also providing mental health services. This includes managing stressors, increasing life skills, and developing meaningful links to needed services. Case managers provide individual, group, and family counseling, facilitate linkages to primary care, and increase social support. The effectiveness of the program was assessed through a Logic Model, Theory of Change, Feasibility Assessment, Evaluation Design with Measures, and proposed Budget and Timeline.
Poster 169: Evaluating Standards-Based Reform: Methods and Outcomes
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Hannah Gbenro, Seattle University, pennah@seattleu.edu
Robert Hughes, Seattle University, rhughes@seattleu.edu
Abstract: Standards-based reform is currently a significant movement in the field of educationTeachers, site-based leaders, and district leaders across the country are experiencing significant changes as they continually strive to create the best educational opportunities for students. As educational practices proliferate, it will be critical to create concurrent models of evaluation that address the unique needs of standards-based reform practices. This evaluation identified the critical questions, and challenges that must be explored in similar evaluations. The presentation will discuss how the findings of this evaluation suggest specific areas that should be evaluated focusing on successes and barriers related to standards-based reform.
Poster 225: A Participatory Approach to Understanding Program Sustainability
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Amy Gedal Douglass, George Washington University, agdouglass@gwu.edu
Freema Davis, Global Family Village Inc, freema.davis@gmail.com
Abstract: Global Family Village (GFV) is a sustainable orphan care model. GFV provides funding and technical support to local partners implementing their model. As part of the model's sustainable design, the partner is expected to completely take over financial and operational responsibility of implementation after 5-7 years. In Bungamati Village, Nepal, over the last two years, GFV and a local partner have successfully implemented the major program components of the model; however, it remains unclear whether the program is on track for sustainability. GFV, in collaboration with its local partner, conducted a participatory evaluation to understand the meaning and process of sustainability from the perspective of local stakeholders. Through a qualitative evaluation including interviews with local leaders and observations of community activities, we identified ways to enhance the likelihood of achieving sustainability and came to a shared understanding of the model and the concept of sustainability.
Poster 185: Examining the Impact of Health Information Technology in Latino Communities
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Miryam Gerdine, HIMSS Latino Community, miryamgerdine@gmail.com
Abstract: Research shows mixed evidence in technology adoption among providers who care for Latino communities as compared to other providers. Some findings suggests that Federally Qualified Health Centers (FQHCs) who treat large percentages of Latino communities have relatively robust adoption of EHRs due to the Health Center Controlled Networks (HCCNs) program and other initiatives. The two case studies examined in this poster are "Using telepsychiatry for the treatment of depression in underserved Hispanics" and "Using health IT to improve quality, coordination, and access for rural and migrant populations." Both of these case studies are part of the case studies for the congressionally mandated report, "Understanding the Impact of Health IT on Underserved Communities and those with Health Disparities." The examination of these case studies have identified encouraging examples of how community-based efforts to incorporate health IT into interventions that drive better health outcomes for Latino families and communities.
Poster 152: Improving the Usability of the Concerns-Based Adoption Model: Validation of a Revised Diagnostic Tool for Measuring Levels of Use
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Susan Gillmor, University of Kansas, scgillmor@ku.edu
Kayla Carter, University of Kansas, kaylasuponcarter@ku.edu
Abstract: This study validates a new scale used to measure Levels of Use (LoU). While LoU has been shown to be an important factor in explaining effectiveness of innovative interventions, few evaluations take advantage of this informative measurement because the interview tool is lengthy and time consuming. This study tests the construct validity of a newly developed Levels of Use survey tool. A random sample of 50 of the nation's nearly 700 child advocacy centers have been solicited to participate in a LoU phone interview about their center's evaluation practices. In addition to the phone interview, the child advocacy centers will receive the Levels of Use scale my email to be completed. The results of the two tests will be compared using a repeated measures analysis of variance to investigate the validity of the shortened scale.
Poster 41: Comparative Power of thew ANOVA, Approximate Randomization ANOVA, and Kruskal-Wallis Test
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Jamie Gleason, Wayne State University, jamie.gleason@wayne.edu
Shlomo Sawilowsky, Wayne State University, professorshlomo@gmail.com
Abstract: The F statistic is said to be robust to departures from normality, though that does not necessarily make it the best test for every situation. The randomization ANOVA presents as an alternative to the ANOVA under non-normal conditions to rehabilitate loss of power. Researchers suggest that the randomization ANOVA is superior to rank tests like the Kruskal-Wallis because ranking data disposes of valuable information. The researchers performed a Monte Carlo analysis on group sizes of n=10 to n=30 and groups of k=3 and k=5 using Fortran. The researchers implemented equal treatment effect sizes of small (0.1-â) to huge (1.0-â) on each group, until all but one group had received a treatment. Data were drawn from the normal, uniform, and chi-square (df=2) distributions. The ANOVA and approximate randomization ANOVA exhibited equal power universally. The Kruskal-Wallis was slightly less than the others in normal conditions and vastly more powerful under non-normal conditions.
Poster 130: "Welcome to the Den of Learning! I love Biology!" Encouraging Inquiry Based Learning in Environmental Science
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Barbara Goldberg, Barbara Goldberg & Associates LLC, barbaragoldb@gmail.com
Rekha Shukla, Barbara Goldberg & Associates LLC, shukla.rekha@gmail.com
Abstract: The Science Education Partnership Award (SEPA) program helps high school science teachers get more students involved in doing classroom-based research with an environmental health focus. Using inquiry-based modules developed at University of Wisconsin Milwaukee, teachers and students study the effects of various environmental agents on the development and behavior of organisms such as fathead minnows, zebrafish and earthworms. They communicate their research via the scientific process of writing papers and creating scientific posters. The yearly activity culminates in a spring research conference. Students responded to surveys before and after their work with the zebrafish module. Based on analysis of 244 matched respondents, the evaluation team found that students demonstrated increased knowledge about the effects of toxicants on zebrafish and were able to extrapolate how these effects translated to similar effects on human embryonic development. Students further indicated greater interest in pursuing a career in environmental science.
Poster 211: Using a Longitudinal Approach to Measuring Parent Engagement in a Home Visiting Program
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Samantha Goodrich, University of Connecticut, samantha.goodrich@uconn.edu
Courtney Lincoln, University of Connecticut, courtney.lincoln@uconn.edu
Alicia Leland, University of Connecticut, alicia.leland@uconn.edu
Katherine Bryce, University of Connecticut, katherine.bryce@uconn.edu
Preston Britner, University of Connecticut, preston.britner@uconn.edu
JoAnn Robinson, University of Connecticut, joann.robinson@uconn.edu
Anne Farrell, University of Connecticut, anne.farrell@uconn.edu
Abstract: Parent engagement in services has become a key process variable for evaluators as they begin to understand how program outcomes differ for sub-groups of participants. However, parent engagement is defined and measured in a wide variety of ways, which it is partly dependent on the format and expectations of the program. The current study explores engagement in a home-based early intervention program. In particular, this study aimed to go beyond dosage indicators of engagement to capture the degree to which parents collaborate with and are actively involved with the home visitors. To further define active involvement and to explore patterns in engagement over time, this study tracks four qualities of engagement longitudinally: parent disposition during the visit, parent insight during the helping process, learning moments, and the parents' progress towards goals. The presentation will discuss the measurement of engagement in services and the feasibility of examining it longitudinally.
Poster 180: Using Accountability Data to Drive Decision Making for Master Tobacco Settlement Monies
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Tracy Graybill, University of Kansas, tgraybill@ku.edu
Rebekah Meek, University of Kansas, r364m036@ku.edu
Karin Chang-Rios, University of Kansas, kcr@ku.edu
Abstract: This poster will demonstrate how the state of Kansas has applied the Children's Initiative Accountability Framework to the evaluation of its Children's Initiative Fund. The state of Kansas uses the data provided by this framework to drive informed decision making regarding the funding of early childhood programs through the Tobacco Master settlement. The Children's Initiative Accountability Framework was developed in 2006 by the Institute for Educational Research in Public Service at the University of Kansas and reflects current trends in accountability. The process meets the requirements of Kansas statutes to use best practices in the field, data to benchmark outcomes, and an evaluation process determining program performance. This poster will highlight the last five years of accountability data and how it has been tied to funding decisions. The poster will be useful to evaluators in understanding how evaluation and funding decisions are made in a politically driven environment.
Poster 134: All Together Now: How Challenges Became Opportunities While Implementing and Evaluating an Accountability System in an Education Nonprofit Network
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Megan Renee Walker Grimaldi, Communities In Schools, meganreneewalker@gmail.com
Abstract: During the previous 6 years, Communities In Schools (CIS), an education nonprofit with nearly 200 affiliates across the United States, created a Total Quality System (TQS) of standards to achieve uniform quality in business practice and program implementation across the CIS network. To ensure that all affiliates were adhering to these standards, the CIS national office began conducting accreditation reviews of affiliates. Researchers at the national office used qualitative analysis of interview data collected during reviews to discover network trends and evaluate the impact of TQS. In this poster, researchers will a) present an overview of findings from interview data; b) describe the challenging transition from accrediting affiliates to evaluating the broader impact of TQS; c) describe the steps taken to encourage use of the findings in a network of diverse talents and interests; and d) evaluate attempts to encourage use of the findings and propose suggestions for similar efforts.
Poster 138: Evaluating Cultural Competence of Community Health Workers
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Andrea Guajardo, University of the Incarnate Word, adguajar@student.uiwtx.edu
Abstract: The Community Health Worker (CHW) model is accepted as an effective model for health promotion and prevention strategies such as immunization, screenings, and long-term monitoring of health outcomes. Examples of successful CHW interventions are reported and cite cultural competence as essential to the success of the model (Henderson, Kendall, & See 2010); however, limited evaluation has been performed to measure the extent of CHW cultural competence or to describe culturally competent behaviors among CHWs. Core components of cultural competence have been identified (Jirwe, Gerrish, & Keene, 2008) and have been measured in nurses, physicians, and other health professions (Schim, Doorenbos, & Borse, 2005,) but the same measurement among CHWs is not widely reported. Further work is necessary to ensure that the linguistic and behavioral characteristics of the CHW are defined in order to enhance curricula used to train CHWs and to ensure implementation of learned behaviors into operational practice.
Poster 58: Teaching Outpatient Substance Abuse Clinicians New Addictions Research; Results of a Novel Perceptions Measuring Instrument Among 40 Clinicians
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Laith Gulli, Private Practice Family Therapist, lfgulli@gmail.com
Nicole Mallory, Private Practice Physician Assistant, nicolemariemallory@yahoo.com
Linda Hesson, Private Practice Psychologist, admin@newlightconsultants.net
Abstract: An originally devised 13-item Likert type-scale measuring instrument with an internal consistency indicated by Cronbach's alpha of .814 was used in the current study to measure and evaluate specific perceptions pertaining to treatment usage with patients and therapist's continuing education and knowledge of new addictions research. The measuring instrument was dispersed to addiction counseling professionals in six clinics located in five different cities within the State of Michigan, with a response rate of 53.3% (N=40). Most of the respondents either strongly agreed or agreed (40% and 27%) when answering a question if treatment compliance would improve and treatment completion rates would increase (27.5% and 35%), if patients were provided with a psychoeducational intervention involving new addictions research. Addictions professionals reported that they had personal financial difficulty paying for new addictions research education (82.5%) and there was a lack of government subsidized education (77.5%) specific to new addictions research.
Poster 240: Engaging Your Data: Using Process Success Stories to Promote Program Achievements and Report Interim Results
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Bernadette Guzman, University of Louisville, bmguzman21@gmail.com
Rene Lavinghouze, Centers for Disease Control and Prevention, shl3@cdc.gov
Pat Rieker, Boston University, rieker@bu.edu
Shauntrelle Andrews, Centers for Disease Control and Prevention, scandrews@cdc.gov
Abstract: Public health practitioners often believe the data speak for themselves. Simply producing data is insufficient unless it can be engaged. Therefore, engaged data is necessary for building understanding as it can foster commitment and passion among stakeholders. Engaged data, as defined through the Component Model of Infrastructure, needs to be communicated early, often, and in a timely manner (Lavinghouze, Snyder, Rieker, 2013). The Center for Disease Control and Prevention's (CDC) Office on Smoking and Health (CDC/OSH) evaluated state programs funded through Communities Putting Prevention to Work (CPPW). Due to the complexity and short time span for the initiative, this was a challenge when outcome data were not immediately available (CDC 2007). CDC/OSH utilized success stories to effectively describe programmatics, accomplishments, and interim results. Following both Made to Stick and Impact and Value: Telling your Programs Story (CDC, 2007) methods, OSH collaboratively created success stories with several state awardees.
Poster 109: Catholic Relief Services Guidance on Monitoring and Evaluation: A Hands-on Resource for Improving Monitoring and Evaluation (M&E) Practice
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Clara Hagens, Catholic Relief Services, clara.hagens@crs.org
Abstract: Catholic Relief Services Guidance on Monitoring and Evaluation (M&E) has supported project-level teams to engage with M&E topics, ranging from gender in M&E, random and purposeful sampling, and data collection tool design to community participation in M&E, evaluation practice, and M&E in an emergency response. The Guidance presents standards of good practice for each topic and an accompanying narrative describing how to operationalize the standards. The Guidance also includes checklists, tips, tool examples and planning tables to support M&E practitioners and non-practitioners alike to improve the quality of M&E system design, implementation and use.
Poster 205: Pragmatism and its Relevance to Evaluation: A Linking Framework
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Jori Hall, University of Georgia, jorihall@uga.edu
Abstract: Pragmatism is often thought of as a companion to evaluation because of its focus on applied inquiry. Unfortunately, pragmatism is often misunderstood as a "whatever works" approach to evaluation rather than a philosophical perspective including paradigmatic assumptions. Understanding philosophical underpinnings are important to inform the theoretical perspectives and practical strategies related to evaluation practice. Accordingly, this work clarifies the philosophy of pragmatism, and its relevance to evaluation practice. In particular, Deweyan pragmatism will be discussed in terms of its key underlying assumptions. The extent to which Dewey's thoughts on pragmatism can be used as an overarching meta-framework that links key theories related to evaluation practice and benefit evaluation practice will also be explored.
Poster 146: Data Collection and Displays in a Universal Design for Learning (UDL) Digital Tool that Supports Middle School Students' Writing
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Tracey Hall, CAST, thall@cast.org
Kristin Robinson, CAST, krobinson@cast.org
Abstract: Data can guide classroom-wide instruction as well as individual interventions (Deno, 2003, Tindal, Ketterlin-Geller, & Alonzo, 2007). In this poster presentation we will highlight how CAST has integrated visual displays of formative assessment using Curriculum-Based Measurement (CBM) into a web-based tool to support middle school students in writing. We will illustrate how CBM results are displayed, and how users receive graphic displays of each measure immediately, which enables students, teachers, and parents to literally see student performance via those visuals. The immediacy and visual representation of results helps students involved in monitoring their progress to be more knowledgeable about their own achievement (Fuchs, Deno, & Mirkin, 1984).
Poster 170: Beyond Methods and Design: Building Organizational Capacity to Participate in Randomized-Controlled Trial (RCT) Evaluations
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Rachel Hallum-Montes, Cicatelli Associates Inc (CAI), rhallum@ufl.edu
Jennifer Rose, Cicatelli Associates Inc (CAI), jrose@caiglobal.org
Edward Rowlands, Cicatelli Associates Inc (CAI), erowlands@caiglobal.org
Abstract: This presentation reports on strategies and lessons learned in providing evaluation technical assistance for foster care agencies participating in an RCT evaluation of a teen pregnancy prevention program in New York City. Strategies and lessons learned are discussed in relation to completing project groundwork prior to project implementation, including: conducting organizational capacity assessments with implementing agencies to identify where areas where technical assistance is needed; becoming familiar with agency regulatory review requirements; securing the proactive support of agency leaders for the evaluation; and working with agency staff and leadership to develop a plan with clearly defined roles and responsibilities for recruiting and retaining evaluation participants. Additional lessons learned pertain to: developing plans for training new agency staff to address problems of high turnover; and maintaining regular, frequent contact between agencies and providers of evaluation technical assistance to rapidly identify and address problems that may arise.
Poster 75: Validating Program Theories: A New Role for Qualitative Data Using Amazon's Mechanical Turk
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Elena Harman, Claremont Graduate University, elena.harman@cgu.edu
Tarek Azzam, Claremont Graduate University, tarek.azzam@cgu.edu
Abstract: A major barrier to the practice of theory-driven evaluation is the lack of validated social science theories (Weiss, 1997a). If a program is not based on social science theory, evaluators must test implicit theories about a program's operation. To fill this role, evaluators need an arsenal of methodological techniques to validate program theory (Bickman, 1989). Some quantitative validation methods have emerged (Marquart, 1990), but qualitative methods remain underdeveloped. The present study demonstrates a novel methodology for validating program theory using qualitative data. The process combines pattern matching techniques (Trochim, 1989) with a qualitative analysis approach utilizing Amazon's Mechanical Turk. This approach has the potential to improve evaluation practice by proposing a reliable technique for validating program theory using qualitative data, thus strengthening theory-driven evaluation and expanding the range of programs to which it can be applied.
Poster 131: Comparing the Use of In-person and Online Instructional Formats for Training Undergraduate Students in Addiction-related Topics
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Joyce Hartje, University of Nevada, jhartje@casat.org
Nancy Roget, University of Nevada, Reno, roget@unr.edu
Wendy Woods, University of Nevada, Reno, wwoods@casat.org
Terra Hambling, University of Nevada, Reno, thambling@casat.org
Abstract: Implementation of Health Reform will help drive integration of physical and behavioral health. Since numerous studies have shown that nurses, social workers, physicians, healthcare workers, psychologists report feeling unprepared to assist or intervene with patients with/affected by SUDs, it is essential that undergraduate allied health students be exposed early in their academic careers to addiction-related education. In 2011, more than a third of college students nationally reported taking one online academic course. There is an abundance of outcome data demonstrating the efficacy of online academic course instruction. However, limited data exist comparing the effectiveness of in-person vs. online instruction for addiction-related courses. To bridge this gap, a study was conducted comparing effectiveness and student satisfaction between in-person and online undergraduate generalist addiction courses. Approximately 1200 students completed either the in-person or online version of the course during Spring 2011 through Fall 2012 semesters. Complete item analysis will be presented.
Poster 209: A New Instrument for Evaluating Science Learning Experiences: What is Science Learning Activation? How Can We Measure it? Is it Useful for Evaluation?
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Ardice Hartry, University of California, Berkeley, hartry@berkeley.edu
Juna Snow, University of California, Berkeley, jsnow@berkeley.edu
Abstract: This poster has two main purposes: first, to provide information to evaluators about a new instrument, developed by researchers at UC Berkeley, University of Pennsylvania, and SRI, that measures science learning activation; second, to collect feedback through conversations about how this measure could be made useful for evaluations of science learning experiences. The poster will describe the instrument and show how we have synthesized prior research on key dispositions, practices, and knowledge in science learning into five key dimensions that build on large prior literatures and involve novel conceptualizations of what is most important for science learning for youth ages 11-14 years. The poster also provides an opportunity to reflect on how we develop instruments and collaborate across disciplines by asking how this instrument can be made useful for evaluative purposes, shared with non-experts, and disseminated to a broad audience of researchers and evaluators.
Poster 99: iSchool Inclusion Institute: Challenges and Opportunities Assessing the Increase of Diversity Via Longitudinal Survey Methods
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Everett Herman, University of Pittsburgh, elh63@pitt.edu
Dana Winters, University of Pittsburgh, dmw67@pitt.edu
Abstract: This poster will illustrate the methods, challenges, and opportunities evaluators implemented to assess an on-going summer institute for undergraduate students interested in the information sciences. The goals of the iSchool Inclusion Institute (i3) Initiative is to attract and prepare students from underrepresented populations to pursue graduate studies in the information sciences, encourage the most promising students to earn terminal degrees, and ultimately to increase faculty diversity in the information sciences. Evaluators have designed a pre-post survey design to measure outcomes at pivotal times throughout the intervention. Some obstacles evaluators have overcome in the evaluation process are small sample sizes, limited differences between participant ethnicities, and participant mortality. These challenges have been mediated by opportunities of high response rates, disaggregating data by gender, a significant length of time to observe participants, and the ability to expand the evaluation as the program grows or replicates.
Poster 206: Using Post Completion Survey Incentives to Improve Survey Response Rates in a Low Income Population Seeking Safety Net Care
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Susan Hewitt, Health District of Northern Larimer County, shewitt@healthdistrict.org
Jillian Jetter, Health District of Northern Larimer County, jjetter@healthdistrict.org
Abstract: Evaluators are facing almost constant change in the Early 21st Century when surveys are needed to collect data to inform decisions. A method that worked well a year ago may no longer deliver sufficient results as our target populations increasingly adapt to new technologies. In 2012 an internal evaluation team was tasked with evaluating a new web-based lottery that was implemented to fill new patient openings at a high demand community safety net dental clinic. The population served is low income and uninsured. We chose a mixed mail and Internet survey mode that reflected the clinic's new web-based lottery design. With a limited budget for incentives, and very limited recent published studies on incentives and web-based surveys in low income populations, we conducted a small study within the study to evaluate the impact of a small post completion incentive on response rates among a low income, resource limited population.
Poster 92: Evaluating a Gap/Bridge Year Program: Challenges of Self-selected Participants and Evaluating in a New and Unexplored Field
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Nina Hoe, University of Pennsylvania, ninahoe@gmail.com
Shannah Metz, Global Citizen Year, shannah@globalcitizenyear.org
Abstract: The gap/bridge year movement is on the rise in the U.S., and Global Citizen Year, a growing program receiving significant external support, recruits and trains a diverse corps of high-potential graduating seniors and immerses them in emerging economies during a "bridge year" between high school and college. This paper will focus on the challenges associated with being the first known gap/bridge year program to conduct and connect a process and outcome evaluation using both surveys and participant interviews, and share its findings with both funders and within the industry. Specifically, the paper will focus on and the ways in which the evaluators attempted to (1) overcome issues of self-selection and the absence of national or "norm" data on gap/bridge year participants or programs, (2) understand evaluation results in a new and evolving organization and field, and (3) connect learning outcomes to program elements.
Poster 145: Conducting a Health and Wellness Survey With Public Sector Employees: Lessons Learned
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Angela Horvath, City of St Petersburg, angela.horvath@stpete.org
Abstract: Health promotion programs have grown increasingly popular in the workplace as organizations seek to reduce healthcare costs, absenteeism, and presenteeism, as well as increase employee wellbeing and productivity. As part of the Health and Wellness component of its Succession Plan, a public utility in a mid-sized, urban Florida city sought to capture employee feedback and information on health topics via a voluntary, anonymous survey. An initial questionnaire was developed and contained topics such as healthy behaviors, preventive care, tobacco use, and perceived health status. Focus groups were used to pilot the initial survey with 9% of all employees. Based on feedback obtained from the focus groups, modifications were made and the survey was administered. Issues expressed during the focus groups, such as privacy concerns and question clarity, are presented here.
Poster 208: Evaluating Peers as Knowledge Providers: Evaluation Design in Engineering Project-Based Learning
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Jenna Howard, Georgia Institute of Technology, jhoward63@gatech.edu
Julia Melkers, Georiga Institute of Technology, jmelkers@gatech.edu
Ed Coyle, Georiga Institute of Technology, ejc@gatech.edu
Randal Abler, Georgia Institute of Technology, randal.abler@gatech.edu
Abstract: Student learning in engineering is increasingly geared toward collaborative projects that allow for learning to take place through social spaces. These projects typically involve peer interactions and the creation of social communities that focus on applied engineering problems, important for the development of professional capabilities. Engineering has evolved to embrace the different approaches to formal learning through classroom and applied experiences and informal learning that occurs outside of structured activities. This poster will present a multi-methodological evaluation of student learning in the NSF-funded Vertically Integrated Project (VIP) in the K-12 and Higher-Ed environment. There are inherent challenges in evaluating the interim knowledge-development of large project-based learning groups and in connecting interim outcomes to learning outcomes. This project uses self reported survey data and longitudinal social network analysis together with grade and performance data to answer; how do knowledge networks of students promote the development of STEM skills?
Poster 9: Change is a Two-Way Street: Strategies for Successful Communication Within a Developing Evaluation Process for a Growing Non-profit Organization
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Maisha Howard, Children's Literacy Initiative, mhoward@cli.org
Kristin Haegele-Hill, Children's Literacy Initiative, khaegelehill@cli.org
Megan Barner, Children's Literacy Initiative, mbarner@cli.org
Abstract: Drawing from our experience developing a knowledge management system for internal stakeholders within our organization, we will introduce a series of communication strategies and procedures established in parallel to the standardization and refinement of internal evaluation processes. We will establish how communication can be a pivotal factor in fostering a culture of evaluation within a growing non-profit organization - overcoming resistance and supporting sustainable change.
Poster 48: Using Regression Discontinuity Design in Evaluating the Effectiveness of a Pre-K Program
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Penelope Huang, Applied Survey Research, penny@appliedsurveyresearch.org
Abstract: The usual method of evaluating the effectiveness of pre-K programs typically involves a comparison of mean scores on some school readiness measures between students who attended pre-K and those who did not. Even when a host of covariates is included in a regression model estimating the net effect of preschool attendance, estimates are likely to be biased as a result of selection effects. That is, children who attend preschool are likely to be different from those who do not in non-random, systematic ways that will lead to biased estimates of the effect of preschool. Since preschool attendance cannot be randomly assigned, a quasi-experimental method can be employed to approximate random assignment to reduce the amount of bias due to selection effects. Regression discontinuity design is described in this poster and is used to examine the effect of preschool on early literacy, early math skills, receptive vocabulary, and self-regulation.
Poster 47: Growing by Leaps and Bounds: The Use of Multiple Measures to Inform Child Welfare Outcomes
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Penelope Huang, Applied Survey Research, penny@appliedsurveyresearch.org
Abstract: The Leaps & Bounds (L&B) program is designed to support healthy development of infants and young children whose parents are recovering from methamphetamine use and are participants of Santa Cruz County's Dependency Drug Court program. L&B provides support to parents as they learn about the needs of their children and participate as part of a larger team to address these needs. Parents receive hands-on parent education and assistance in arranging, scheduling, and participating in services with, and for, their children through home visits and intensive interactive therapy. The program is very assessment-heavy and provides multiple measures and time points of measurement to inform as to the effectiveness of the program. Given the needs of the service population and their reluctance to engage in formal assessment, the efficacy of multiple assessments is examined and methods of successful engagement and data collection are considered.
Poster 215: Using Data Mining Techniques to Analyze Differential Program Effectiveness: Classification and Regression Trees (CART) and Random Forests
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Nicholas Huntington, Advocates for Human Potential Inc, nhuntington@gmail.com
Abstract: Stakeholders working in health, behavioral health, and human services interventions often wish to understand differential effectiveness, whether their intervention is helping the full range of clients they serve or whether instead there are sub-groups of clients not being adequately helped. Two related quantitative techniques from the field of data mining: Classification and regression trees (CART), and random forests, provide potentially powerful alternatives to traditional statistical methods for addressing this question. These techniques have several advantages: 1) they are inductive and pattern-finding, 2) they can draw on any number of predictors, 3) they excel at uncovering interactive effects, 4) they provide results in an intuitively understandable tree diagram, 5) they produce targeted, "actionable" results, and 6) they make few requirements of their input data. This presentation describes these techniques and illustrates their application in a human services evaluation. These techniques could fruitfully be used to provide ongoing formative feedback to programs.
Poster 34: A Typology of Terms in Evaluation
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Kylie Hutchinson, Community Solutions Planning & Evaluation, kylieh@communitysolutions.ca
Abstract: The field of evaluation is rife with inconsistent and overlapping terminology that stems from its use in different sectors, funders, geography, and individual program areas. In an effort to provide some clarity for evaluators everywhere, we have developed a Typology Map and Evaluation Glossary mobile app of key terms in the field. Come view both and add your comments and suggestions.
Poster 196: Use of Emergency Department Services for Non-Emergent Conditions Among Adults With Disabilities
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
David Idala, University of Maryland Baltimore County, didala@hilltop.umbc.edu
Nancy Miller, The Public Policy Department at the University of Maryland, Baltimore County, nanmille@umbc.edu
Adele Kirk, University of Maryland, Baltimore County, akirk@umbc.edu
Charles Betley, University of Maryland Baltimore County, cbetley@hilltop.umbc.edu
Seung Kim, University of Maryland Baltimore County, skim@hilltop.umbc.edu
Abstract: An estimated 1 in 10 non-institutionalized adults aged 18 to 64 years in the U.S. experience a disability. Adults with disabilities tend to use ED services more frequently than those without disabilities. The differences in ED use among specific populations may indicate disparities in access to health care. Additionally, the populations that commonly experience negative health disparities are likely to become an even larger proportion of the American population. Health insurance provides an important link to improved health outcomes by means of better access to health care. This research evaluation is intended to explore the differential impact of race/ethnicity and insurance status on ED use among individuals with disabilities. Gaps between particular groups may signal opportunities for future interventions to alleviate the disparities, where good evaluation design will be necessary to assess costs and benefits of interventions directed at disabled populations and particular subsets.
Poster 224: Inducing Empowerment Evaluation: A Case Study at a Japanese University
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Tomoko Kamada, University of Toyama, ktomoko@las.u-toyama.ac.jp
Kazuko Nakagawa, Toyama Japonica, dqh03060@nifty.ne.jp
Yukiko Watanaba, University of California, Berkeley, yukikow@berkeley.edu
Abstract: Higher education in Japan is a relatively privileged context, and conflicts with the goal of empowerment, i.e. to advance the rights and status of those who are socially vulnerable. Japanese university programs themselves do not conduct activities aimed at empowerment as in NPOs activities in developing countries. The authors implemented Empowerment Evaluation (EE) following Fetterman and Wanderman’s (2005) principles in a Japanese university setting where evaluation culture and system remain underdeveloped and empowerment is not amenable to the context. We will (a) introduce consciousness-raising methodologies and strategies that induced empowerment evaluation in our context and (b) report research findings on various factors that enabled and strengthened the implementation of empowerment evaluation.
Poster 116: Evaluating Sense of Community in the Science, Technology, Engineering, and Mathematics (STEM) Transfer Experience
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Jason Katz, University of South Carolina, jaskatz@gmail.com
Abraham Wandersman, University of South Carolina, wandersman@sc.edu
Tim Mousseau, University of South Carolina, mousseau@sc.edu
Greg Townley, University of South Carolina, gregory.townley@gmail.com
Abstract: Undergraduate transfer student literature indicates that the experience of "transfer shock" may influence both social and academic factors. We will share the results of an evaluation study of comprehensive intervention strategies to decrease transfer shock and raise sense of community amongst STEM-majoring undergraduate transfer students. Our poster will include an assessment of how students' sense of community may serve as a protective factor to decrease transfer shock and increase the probability of STEM students' success. students who have been enrolled for more semesters are also more likely to have higher sense of community, which suggests the long-term utility of remaining involved in the program. Also, in STEPs to STEM more than in the other groups, there is a strong relationship between actual sense of community and positive perceptions of the university environment.
Poster 31: Evaluation of the Medical Research Program at the Department of Veterans Affairs
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Luba Katz, Abt Associates Inc, luba_katz@abtassoc.com
Rebecca Fink, Abt Associates Inc, rebeccav_fink@abtassoc.com
Sam Bozeman, Abt Associates Inc, sam_bozeman@abtassoc.com
Abstract: The goal of the medical research program at the Department of Veterans Affairs (VA) is to discover knowledge, develop researchers and health care leaders, and advance health care for Veterans and the nation. To assess the program, we developed a theoretical framework to measure performance on several domains associated with productive research environments, including: (1) organizational strategy and investment; (2) quality and impact of research; (3) collaboration; (4) autonomy and freedom; (5) research resources; (6) support systems; (7) rewards structure and job security; and (8) professional development opportunities. Data for each domain were then collected via a survey of 1,032 VA-funded researchers. The organizational strategy and autonomy to conduct research emerged as the strongest and research resources as the weakest areas in the VA research environment. Association between satisfaction with various research processes and supports and overall program satisfaction was also examined.
Poster 55: Development of the Value-Added Assessment System of Student's Learning Achievement to Monitor and Evaluate the Educational Provision in Basic Education of Thailand
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Jatuphum Ketchatturat, Khon Kaen University, jketchatturat@hotmail.com
Samphan Phanpruek, The National Institute of Educational Testing Service, sampha@kku.ac.th
Abstract: The system of Value-Added Assessment of Student's Learning Achievement has been developed to monitor and evaluate the educational provision in basic education. The steps of development consist of 1) study and analyze the school and the district report system of student achievement and progress 2) collect the data of student achievement to develop the value added indicator 3) develop the system of value-added assessment by Participatory Action Research Approach 4) put the system of value-added assessment into the Educational District of Secondary School 5) determine the quality of the developed system of Value-Added Assessment. The components of the developed system consist of 1) the database of value-Added assessment of student's learning achievement 2) the process of monitoring and evaluation the student's learning achievement and 3) the reporting system of Value-Added Assessment of student's learning achievement.
Poster 248: Evaluating Independent Accountability Mechanisms: Can We Develop Appropriate Standards?
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Werner Kiene, Inter-American Development Bank, wkiene@gmail.com
Isabel Lavadenz, Inter_American Development Bank, 
Alf  Jerve, World Bank, 
Abstract: Independent Accountability Mechanisms (IAMs) have become important features of many institutions. For instance, the World Bank has an Inspection Panel that is directly reporting to the Banks Board of Executive Directors. Other International Finance Institutions (IFIs) have similar units dealing with policy compliance and related issues. The paper will report on several issues that have arisen in recent reviews and evaluations of Independent Accountability Mechanism of International Finance Institutions. It will, for instance, deal with issues of cross-oversight by units that are at the same hierarchical level in a particular organization. It will discuss existing efforts to improve the evaluability of these Mechanisms and it will draw on relevant experiences from similar accountability efforts (audit, institutional integrity units, etc) to suggest an approach that could make future evaluations of Independent Accountability Mechanism more professional and more reliable.
Poster 22: Beyond Basic Metrics: Tapping the Evaluative Power of Social Media
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Robin Kipke, University of California at Davis, rakipke@ucdavis.edu
Abstract: Social media is transforming the way people obtain and share information. Because of this, advocacy organizations need understand how they can make social media work for them. Basic metrics like number of views, followers, and shares amount to little more than an online popularity contest. Social media analytics can be used to reveal so much more! This session will demonstrate how organizations can capitalize on evaluative social media tools like Facebook Insights, HootSuite and Google Analytics to collect data about target audiences, competitors and allies; improve the effectiveness of social media messaging and strategies; and monitor the effects of these campaigns on program information requests, volunteerism, and coalition participation.
Poster 184: Mapping access for Understanding: Implications of Mapping Mental Health Resources in Louisiana for Program and Evaluation Outcomes
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Janna Knight, Louisiana Public Health Institute, jknight@lphi.org
Kris Gerig, Louisiana Public Health Institute, kgerig@lphi.org
Snigdha Mukherjee, Louisiana Public Health Institute, smukherjee@lphi.org
Jessica Diedling, Louisiana Public Health Institute, jessica.diedling@la.gov
Cynthia Suire, Louisiana Public Health Institute, cynthia.suire@la.gov
Amy Zapata, Louisiana Public Health Institute, amy.zapata@la.gov
Abstract: As evaluators of Louisiana's home-visiting program with an Infant Mental Health component, we have received feedback from nurses that clients in some regions of the state have difficulty identifying or accessing mental health services for low-income pregnant women and their young children. A key aspect of current evaluation activities includes tracking when, where, and why home visitors make referrals to external mental health resources. To better understand the context of these referrals, we are mapping mental health service providers and resources using GIS software. We define mental health service providers as those who offer psychotherapy, medication management, and case management to pregnant women, and women and/or their children up to 5. The maps created will be resources for Louisiana nurse home visitors, will support the argument for expansion of mental health services for pregnant and parenting women in underserved regions, and will inform current and future evaluation activities.
Poster 182: Looking Back to Build Forward: A Home Visiting Program Uses Historical Data to Inform Future Directions
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Janna Knight, Louisiana Public Health Institute, jknight@lphi.org
Paula Zeanah, Tulane University, paula.zeanah@la.gov
Snigdha Mukherjee, Louisiana Public Health Institute, smukherjee@lphi.org
Maggie Gustafson, Louisiana Public Health Institute, mgustafson@lphi.org
Jessica Diedling, Louisiana Office of Public Health, jessica.diedling@la.gov
Cynthia Suire, Louisiana Office of Public Health, cynthia.suire@la.gov
Amy Zapata, Louisiana Office of Public Health, amy.zapata@la.gov
Abstract: Louisiana, like many states, has challenges ensuring access to mental health services for low income mothers and their infants. To address this concern, Louisiana's nurse home-visiting program serving first-time pregnant women and their children up to age 2 added a home-visiting Infant Mental Health Consultant (IMH-C) to some of their nurse teams. In order to understand the patterns of referrals to IMH-Cs by program nurses, we will examine nurse referrals (N=177) for the time frame January 2011 to April 2012. Measures will include: Edinburgh Depressions scores; patient report of mental health diagnoses and current treatment; participant demographics; and completed referrals. The results of this analysis will help us to better characterize the IMH-C role in the home-visiting team, understand the issues facing program participants, illuminate how nurses currently practice referral processes in the home visiting context, and will help to inform program approaches in the future.
Poster 88: Pilot testing an Evaluation plan for a School-based Art Therapy Intervention for rEfugee Students From Burma
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Sarah Kowitt, University of North Carolina at Chapel Hill, skowitt@gmail.com
Diane DeTrizio, University of North Carolina at Chapel Hill, ddetrizio@gmail.com
Dane Emmerling, University of North Carolina at Chapel Hill, dane.emmerling@gmail.com
Claire-Helene Mershon, University of North Carolina at Chapel Hill, clairehelene.mershon@gmail.com
Kristin Linton, Art Therapy Institute, kblinton@gmail.com
Hillary Rubesin, Art Therapy Institute, hrubesin@gmail.com
Christine Agnew-Brune, University of North Carolina at Chapel Hill, cbrune@live.unc.edu
Geni Eng, University of North Carolina at Chapel Hill, eugenia_eng@unc.edu
Abstract: In designing an evaluation of an art therapy program for refugee youth from Burma, special considerations emerged beyond selecting culturally congruent instruments. We present the methods and insights gained from pilot testing the overall evaluation plan for such a program offered by the Art Therapy Institute (ATI). Since 2009, ATI and a local school system in North Carolina have collaborated to provide art therapy to recently arrived students from Burma. The three goals of this four-week pilot were the following: assess procedures for administering five validated mental health and academic measurement tools, determine the organizational capacity needed to sustain evaluation activities, and use findings to finalize an evaluation plan for subsequent years. Preliminary findings and resulting changes to protocols are reported. The discussion includes lessons learned for practitioners and evaluators on the benefits of pilot testing evaluation plans for mental health programs delivered by small non-profit organizations to refugee youth.
Poster 24: Issues in Applying Developmental Evaluation: Opportunities and Challenges Found in Early Childhood Development and Extended Care for Seniors in Alberta Projects
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Eugene Krupa, University of Alberta, ekrupa@ualberta.ca
Katharine Woodman, Catalyst Research & Development, kwoodman@catalystrandd.com
Leslie Gardner, , 
Ilene Fleming, , 
Mary Stewart, , 
Poster 212: Monitoring and Evaluation Aspects in The Ukrainian Civil Society Organizations
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Volodymyr Kuprii, CCC Creative Centre, kuprii_phd@ukr.net
Lyubov Palyvoda, , office@ccc.kiev.ua
Abstract: The Ukrainian civil society organizations (CSOs) participating in conducting monitoring and evaluation. Types of monitoring and evaluation can be divided into the following groups: monitoring of projects/programs, monitoring of governmental/municipal program; evaluation of organizational development, projects/programs, public policies. The authors analyze how monitoring and evaluation are incorporated in the activities of the Ukrainian CSOs.
Poster 63: Evaluation Capacity Building: Applying a New Model
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Candace Lacey, Nova Southeastern University, lacey@nova.edu
Abstract: This presentation looks at one non-profit organization's attempt to build capacity by requiring all of their grantees to conduct a learning evaluation. Each potential grantee was required to attend a mandatory training where they were taught to design a logic model to include in their funding application. In addition, regular grantee meetings and one-on-one site visits were conducted by the evaluator to teach the grantees about data collection. The experiences shared in this presentation reflect the evolution of the recognition by the grantees of the impact data can have on grass roots organizations. It can be a means to help them understand their program outcomes, it can help them identify and modify weaknesses, and it can make them aware of the importance of using data to apply for additional funding.
Poster 254: What Makes a School Nurse Program Successful? Cost Savings, Health Benefits or Grades?
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Melinda Landau, San Jose Unified School District (SJUSD), melinda_landau@sjusd.org
Ashini Srivastava, Stanford University, ashini@stanford.edu
Eunice Rodriguez, Stanford University, er23@stanford.edu
Abstract: With increasing budget cuts rigorous evaluation is needed to document the impact of school nurses on student health and academic outcomes. A quasi-experimental evaluation was used to compare four schools with added full-time nurses with five schools with part-time nurses in San Jose Unified School District. Results suggest that when a full-time nurse was added to demonstration schools, the number of absences due to illness dropped. Students with asthma in demonstration schools were less likely to visit hospital emergency departments for asthma related events, and narrowed the academic gap in scoring advanced or proficient on standardized tests when compared with students with no health conditions. Savings due to reduction in absenteeism and lower utilization of emergency department services affirms the need to support school health services. Yet, school administrators might prioritize grades, and there might be a lack of consensus on how strong the findings should be to inf luence change.
Poster 231: The Evaluation of a Brazilian program of Internalization and Retention of Medical Professionals in Sites of Vulnerability as Management Tool
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Aline Leal, Fiocruz Foundation, leal.aline@gmail.com
Catia Martins de Oliveira, Fiocruz Foundation, catia@ensp.fiocruz.br
Marly Cruz, Fiocruz Foundation, marly@ensp.fiocruz.br
Solamge Kanso, Fiocruz Foundation, solange.kanso@gmail.com
Raquel Torres, Fiocruz Foundation, raquelmct@gmail.com
Silvia Cardoso, Fiocruz Foundation, silviac_carvalho@yahoo.com.br
Antonio De Lima, Fiocruz Foundation, antonioniteroi@hotmail.com
Victor Grabois, Fiocruz Foundation, victorgrabois1@gmail.com
Abstract: The Brazilian Government has been facing difficulties in the effectiveness of public policies of internalization and valorization of health professionals, particularly physicians. The Valorization Program of Primary Care Professional (Programa de Valoriza+ºão do Profissional da Aten+ºão Básica - PROVAB) was implemented in 2011 with the aim of investing in strategies for recruiting and qualifying doctors in primary care, offering working conditions both physical and financial, and access to in-service training, as well as the integration of community-service-teaching. The first phase of the evaluation of performance conducted developed a strategic and logic analysis of the program using documentary analysis, secondary data and Webportfolio data. As preliminary results, we identified that some of the strategies adopted were successful, such as the provision of professionals in areas of great vulnerability. However, in-service training and the provision of adequate working conditions were also identified as restructuring points of the program.
Poster 111: Retooling Michigan's Child Support Enforcement: Using Predictive Modeling and Supportive Methods in Michigan's Child Support System to Increase the Financial Well-Being of Children and Self-Sufficiency of Families
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Laura Lein, University of Michigan, leinl@umich.edu
Sue Ann Savas, University of Michigan, ssavas@umich.edu
Jeremy Gaertner, University of Michigan, gaertjl@umich.edu
Laura Sundstrom, University of Michigan, sundstrl@umich.edu
Desiree Liwosz, University of Michigan, liwoszd@umich.edu
Peter Howell, University of Michigan, pkhowell@umich.edu
Erin Sehnert, University of Michigan, esehnert@umich.edu
Abstract: In a 3-year grant period, the Retooling Michigan's Child Support Enforcement Project will design, pilot and evaluate programs to increase the financial well-being of children and the effectiveness of the child support collection process through case stratification research, predictive models, evaluation of existing approaches, and the development and evaluation of innovative pilot collection approaches. The project outcomes expected from the grant-supported activities include, but are not limited to, (1) Improved research on current data to support evidence-driven selection of approaches to child support collection, (2) Sophisticated development of a "tool kit" of current strategies and new piloted strategies, and (3) Dissemination of both research and successful pilot strategies to enable greater success in child support collection. These activities are expected to increase child support collections, increase collections of arrears, and ultimately improve the financial well-being of children and self-sufficiency of families.
Poster 4: Methods and Findings From a Policy Implementation Evaluation: Examining New York City Group Child Care Regulations
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Catherine Lesesne, ICF International, clesesne@icfi.com
Sarah O'Dell, ICF International, sarah.odell@icfi.com
Robert Stephens, ICF International, robert.stephens@icfi.com
Thearis Osuji, ICF International, thearis.osuji@icfi.com
Jan Jernigan, Centers for Disease Control and Prevention, jjernigan1@cdc.gov
Laura Kettel Khan, Centers for Disease Control and Prevention, ldk7@cdc.gov
Laura Lessard, Arcadia University, lessardl@arcadia.edu
Jakub Kakietek, ICF International, jkakietek@icfi.com
Abstract: In 2007, the New York City Department of Health and Mental Hygiene (NYC DOHMH) implemented regulations governing standards on beverages, physical activity and screen time for all licensed group child care centers. In 2009-2010, an evaluation was conducted to explore the level of compliance with the regulation components post implementation and identified characteristics of child care centers associated with compliance. The evaluation also examined if varying degrees of compliance with the regulations was related to child-level outcomes such as level of physical activity while in center care. Thisposter will 1) describe the design employed and methodological considerations for the evaluation; 2) illustrate different methodologies used to measure compliance in group child care centers and the implications of these from an analytic perspective; and 3) share evaluation findings related to if and how compliance was associated with child beverage and physical activity outcomes.
Poster 189: Diffusion of Treatment: Preliminary Mapping of the Cohort-level, Extra-program Impact of CTTI(Chicago Teacher Transformation Institutes), a Science, Technology, Engineering, and Mathematics (STEM) Professional Development Program
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Jonya Leverett, Loyola University Chicago, jlevere@luc.edu
Megan Deiger, Loyola University Chicago, mdeiger@luc.edu
Abstract: CTTI, a STEM professional development (PD) program with an emphasis on teacher leadership in addition to content knowledge and pedagogy has had 211 unique participants from approximately 20 different schools over the past three years. Because teacher mobility is a reality of Chicago Public schools, it is prudent for this program and others like it to understand its impact at the cohort level. Appropriating some of the terminology and methodology of epidemiology ("disease tracking"), this preliminary work will begin mapping teacher/cohort movement and diffusion of treatment beyond the immediate reaches of the program. Existing program data will begin to parse conditions for diffusion, diffusion levels (intra-program, inter- and intra-school, person to person) and also transmission conditions (relationship type and strength, availability of resources for knowledge transmission, knowledge "superspreaders"). Development of this methodology presents a possible tool for evaluators who wish to report on extra-program impact.
Poster 247: A Proposed Model for Health Communication Initiatives Using Traditional and Non-traditional Metrics
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Elyse Levine, Booz Allen Hamilton, levine_elyse@bah.com
Jodie Abbatangelo-Gray, Summit Research Associates NYC, jodie@sra-nyc.com
Amy  Mobley, University of Connecticut at Storrs, amy.mobley@uconn.edu
Grant McLaughlin, Booz Allen Hamilton, mclaughlin_grant@bah.com
Jill Herzog, Booz Allen Hamilton, herzog_jill@bah.com
Abstract: Health communication initiatives for the general public such as Lets Move! and MyPlate present challenges for measuring effectiveness, especially when long-term outcomes, like reduced incidence of overweight and obesity, tend to be considered the most valid indicators. Advancements in frameworks and models for behavior change, coupled with rapid changes in communication technology, provide an opportunity to consider new approaches to evaluating the effectiveness of health communication initiatives. An evaluation framework for general population health communication initiatives is presented that builds on well-known and under-utilized models and theories to propose a wide breadth of observations, outputs, and outcomes that can contribute to a fuller assessment of effectiveness. It is hoped the framework will help identify where research is needed to fill gaps in knowledge, and increase understanding among all stakeholders on the strengths of communication in health initiat ives.
Poster 161: The Cancer Care Collaboration: A Utilization-Focused Approach to Evaluating a Pilot Program Addressing the Psychosocial Needs of People Affected by Cancer
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Holly Lewandowski, Evaluation for Change Inc, hlewandowski@yahoo.com
Lisa Kolavennu, Wellness House, lkolavennu@wellnesshouse.org
Valerie Piazza, Wellness House, vpiazza@wellnesshouse.org
Dawn Williams, Wellness House, dwilliams@wellnesshouse.org
Abstract: Research suggests that more than 50% of people with cancer suffer from psychosocial distress, yet supportive care for patients is not regularly included in traditional cancer treatment. Guidelines for evaluating distress were developed by the National Comprehensive Care Network (NCCN), yet these guidelines have not been widely adopted. By 2015, the American College of Surgeons' Commission on Cancer will mandate routine distress screening and referral for treatment. Cancer centers must meet Cancer Program Standards to be approved as an accredited cancer care center. To meet accreditation requirements, cancer centers that do not have trained staff to address psychosocial needs or the resources to hire staff are collaborating with community based organizations that specialize in psychosocial supportive care. This poster will illustrate findings from an evaluation of a pilot program led by a hospital and a community-based organization (CBO) to provide psychosocial support to cancer patients.
Poster 38: Meeting the Challenges in Cluster-Based Trials in Schools: Lessons From the Cluster-Randomized Trial of Positive Action
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Kendra Lewis, University of California at Davis, kelew@ucdavis.edu
David DuBois, University of Illinois at Chicago, dldubois@uic.edu
Peter Ji, Adler School of Professional Psychology, pji@adler.edu
Joseph Day, Governors State University, jday2@govst.edu
Naida Silverthorn, University of Illinois at Chicago, naida@uic.edu
Niloofar Bavarian, University of California, Berkeley, nbavarian@berkeley.edu
Samuel Vuchinich, Oregon State University, vuchinis@oregonstate.edu
Alan Acock, Oregon State University, alan.acock@oregonstate.edu
Margaret Malloy, Oregon State University, malloym@onid.orst.edu
Marc Schure, Oregon State University, schurem@onid.orst.edu
Brian R Flay, Oregon State University, brian.flay@oregonstate.edu
Abstract: Longitudinal, cluster-randomized controlled trials (CRCTs) can present a multitude of challenges. This paper discusses the design, analyses, challenges, and strengths of the longitudinal Chicago CRCT of Positive Action, a social-emotional and character development program. Challenges with the trial included recruitment, especially to the control condition; consent rates, especially as students aged; finding or developing questionnaires developmentally appropriate for students ages 8-14; small sample size at the school (cluster) level; implementation/fidelity; and student mobility. Strengths of the trial included the eight-waves of data collection across six years; retention of all clusters; supplemental school-, teacher-, and parent-reports of student behaviors; and the use of novel methodologies to address mobility. Lessons learned from this study should prove useful to future evaluators of longitudinal CRCTs.
Poster 124: Utilizing Web-Based Assessment Data to Evaluate a College Freshman Retention Program
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Wenshu Li, The University of Tennessee, wli23@utk.edu
Richard Bennett, The University of Tennessee, rbennet2@utk.edu
Abstract: The College Freshman Retention Program aims to enhance the retention of freshman and prepare them for advanced engineering courses. This paper will address an intervention, homework bonus policy, and its' impact on students' homework and assessment performance overtime. The instructional technology this program used kept all of the records of students' assessment data of homework, lecture attendance, and exams on the course website. The paper will review the process of conducting an evaluation of this STEM initiative and discuss the results of our initial evaluation utilizing the web-based assessment data.
Poster 40: Moving From Silence to Dialogue:The Results of Intergroup Dialogue
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Mayra Lopez, Nyack College, mayra.humphreys@nyack.edu
Abstract: In evaluative, quantitative studies the importance of describing a program can often be overlooked. Evaluative studies can place much of its resources on recruiting participants and operationalizing outcomes, and consequently the intervention itself is minimized. Exploration of a program provides evidence for in-depth, precise descriptions of observations. Programs descriptions also provide clarity as to how, if, or why, program goals and objectives were achieved. Equally important, program descriptions increase knowledge about human service provision and best practices for program implementation. Given the value and benefits of program descriptions, this presentation includes a in-depth description of the Intergroup Dialogue program and its outcomes. Intergroup dialogue is a group model that is intended to help facilitate better group relations.
Poster 121: The Changing Role of the Evaluator: Exploring the Future of Small Evaluation Firms
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Catherine Luke, Duquesne University, clukeint@gmail.com
Christina Luke, State University of New York at Albany, cluke@measinc.com
Abstract: The role of the evaluator can change throughout the course of a program evaluation. In the case of small, independent evaluation firms, there is a pressure to do more: provide more useful feedback, more technical assistance, more capacity-building, more recommendations. How equipped are we to meet those demands? Many evaluators have had to straddle the divide between presenting the evidence and theories of an expert researcher and framing findings so that stakeholders can make sense of the data. This paper explores the changing roles of small-scale evaluators as they seek to draw upon both research evidence and client needs to create the most useful evaluation products.
Poster 43: Monitoring HIVDR by Earning Worning Indicators (EWI)
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Ndiaye Maguatte Ndoye, Ministère de la Santé, guettma@gmail.com
Niang Cheikh, Ministère de la Santé, ngcheikh@gmail.com
Ngom Ndeye Fatou, Ministère de la Santé, ndeyetouti98@yahoo.fr
Souleyman eGning, Ministère de la Santé, bskgning@hotmail.com
Abstract: The emergency of HIV drug resistance (HIVDR) is effective. To maintain the effectiveness of first- and second-line antiretroviral (ARV) regimens and also potentially reducing HIV transmission, WHO recommends that countries develop a national strategy for HIVDR prevention and assessment. The strategy consists to estimate the (EWI) in eligible sentinel ART site should take place in representative sentinel ART sites. This pilot study of estimate EWI was carried during two stages in the twenty site selectioned and the following procedure was : - National HIVDR working groups coordinating is designated, - development of the protocol of research and management tools of data, - Trainning of the sentinel sites and installation tools data record, - Supervision of the sites for management of the data, - Collect, analyse and traitement data Résultats The adapted therapy ART of first ligne is effective among all patients in all sites. However, the performance of the EW2 and EW3 2 is lowered between 2 investigations
Poster 162: Challenges and Triumphs of Assessing Implementation Status and Progress Toward Short-Term and Long-Term Goals: K-12 Initiative for Increasing the Identification of Underrepresented Student Populations in Gifted and Talented Education
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Nyambura Susan Maina, Montgomery County Public Schools, susan_n_maina@mcpsmd.org
Helen Wang, Montgomery County Public Schools, helen_wang@mcpsmd.org
Trisha McGaughey, Montgomery County Public Schools, trisha_a_mcgaughey@mcpsmd.org
Abstract: This poster will cover the evaluation of progress toward short- and long-term goals in implementing a multi-level federal-funded initiative aimed to increase participation of students from underrepresented populations in gifted and talented (GT) education. After presenting program context and the program logic model, the evaluators will focus on methodological approaches used to establish indictors of implementation outcomes and development of appropriate instruments aligned with the components of the logic model with different levels. The evaluators will also detail strategies of multi-site data collection activities targeting different audiences (school administrators, teachers, and students) across a three-year period. Lessons learned with regard to : 1) addressing challenges to collecting quality, consistent, and comparable data across sites; 2) creating online structures to manage multi-site data collection efforts; 3) need for ongoing update of instruments to adequately capture progress over time; and 4) impact of program staff turnover on professional development will be outlined.
Poster 135: Implications of Different Forms of Explanation for the Theory and Application of Logic Models
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Keith Markus, John Jay College of Criminal Justice, kmarkus@aol.com
Abstract: Logic models have a central place in evaluation as a means to represent implementation theories and program theories. Existing literature elaborates many aspects of logic models, but the precise relationships expressed by logic models has not received detailed attention. The present paper explores four illustrative interpretations of the relationships: non-causal explanation (Hempel), real causal explanation (Holland), ideal causal explanation (Pearl) and mereological causal explanation (Markus). Consideration of different explanatory relations can offer the evaluator additional tools for helping to foster evaluability and to offer program descriptions that mesh with how program personnel understand their program. Such distinctions can also help distinguish different evaluation goals and thus help focus the purpose of an evaluation. These distinctions can also help render implementation and program theories more precise and thus help specify methods of testing them. Good evaluation does not require such distinctions, but they offer useful tools for advancing evaluation practice.
Poster 52: Using Social Network Analysis to Measure the Quality and Depth of a Partnership
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
James Marok, University of Toledo, james.marok@rockets.utoledo.edu
Gale Mentzer, Grant Fundamentals LLC, gale@grantfundamentals.com
Abstract: Social network analysis (SNA) can be a powerful tool for the analysis of partnerships. Developing carefully crafted surveys that include not only frequency of interaction but also quality of interaction can be analyzed using SNA to provide rich graphic representations (sociograms) that allow evaluators and stakeholders to easily identify program trends, strengths and weaknesses. A unique use of SNA proposed by the authors allows the evaluator to compare actual sociograms with intended sociograms thereby highlighting process weaknesses and strengths. Viewing a well-designed sociogram can provide a clear picture of relationships although understanding how to depict the sociogram can be elusive. Evaluators must be practiced in thoughtful manipulation of the graphic representations. The way in which sociograms are designed is an important aspect of the analysis. This session will illustrate the difference between useful and confusing sociograms as well as examples of comparisons between actual and intended networks.
Poster 78: Plans, Approaches, Needs, Context, and Reality: Meta-Evaluation of a Portfolio of External Climate Education Projects Funded by the National Aeronautics and Space Administration (NASA)
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Ann Martin, National Aeronautics and Space Administration, ann.m.martin@nasa.gov
Lin Chambers, National Aeronautics and Space Administration, lin.h.chambers@nasa.gov
Margaret Pippin, National Aeronautics and Space Administration, m.pippin@nasa.gov
John Baek, National Oceanic and Atmospheric Administration (NOAA), john.baek@noaa.gov
Abstract: NASA Innovations in Climate Education (NICE) at Langley Research Center has funded 71 climate education initiatives over four years, each evaluated separately by external evaluators. This meta-evaluation project seeks to understand the range of evaluations represented in the portfolio, including descriptive information (what evaluations, questions, designs, approaches, and methods are applied?) and questions of worth (do these evaluations meet the needs of projects and their staff, and of NASA/NICE?). Given the context of climate change education, we also consider the units of analysis and units of change assessed across the portfolio, to determine the extent to which broader questions about community or stewardship behaviors are addressed by such a portfolio. The meta-evaluative rubric was based on key questions related to the Program Evaluation Standards of utility, feasibility, and accuracy; draws from prior meta-evaluation work; and considers the NICE context and the concerns of the environmental and STEM education evaluation communities.
Poster 85: Benefits of Using a Multidisciplinary Approach in Evaluation Research
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Amie Matson, University of Pittsburgh, amiebmatson@gmail.com
Kalani Palmer, University of Pittsburgh, kmp28@pitt.edu
Jennifer Salaway, University of Pittsburgh, jls249@pitt.edu
Abstract: This poster explores the benefits of utilizing a multidisciplinary approach when conducting evaluative research. The University of Pittsburgh Office of Child Development employs a team of professionals who come from diverse schools of thought and whose expertise are shaped by a variety of specialized backgrounds. This poster highlights three projects that exemplify the invaluable use of a multidisciplinary team and the implications for practice. The projects include innovative approaches on multiple levels, including direct interactions with the participants, improvements within the structure of the providers, and changes with policy that directly impact the services being provided. The participants for each of these projects include adolescent females, non-profit organizations, and homeless families with children birth to five. These projects are strengthened by having individuals from various backgrounds work together to evaluate the multiple levels of impact.
Poster 223: Timing for Evaluation Use and Disseminating Products
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Lori Mau, Utah Department of Health, lsugiyam@utah.gov
Abstract: The use of evaluation results is a core principle of evaluation theory and practice, though it remains difficult to apply. Evaluation of the Utah Asthma Program Burden Report provided the opportunity to explore timing and methodology as key components in evaluation use and dissemination of end products. The evaluation employed a sequential mixed methods design using a document review, internet-based questionnaire and key-informant interviews. The evaluation was completed just prior to development of the new burden report and all recommendations on content, formatting, and dissemination methods were utilized. As called for in the evaluation, the burden report was completed in time to be presented to key stakeholders and then released in combination with the Utah Asthma State Plan. This joint release allowed for greater media coverage and improved dissemination to partners. Also, using Google Analytics, key audiences were identified, which led to more targeted dissemination efforts.
Poster 127: Developing a Program-Specific HIV Continuum of Care: An Example From a New York City HIV Housing Program
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Laura McAllister-Hollod, New York City Department of Health and Mental Hygiene, lmcallister@health.nyc.gov
Sarah Braunstein, New York City Department of Health and Mental Hygiene, sbraunstein@health.nyc.gov
John Rojas, New York City Department of Health and Mental Hygiene, jrojas@health.nyc.gov
Mary Irvine, New York City Department of Health and Mental Hygiene, mirvine@health.nyc.gov
Stephanie Chamberlin, New York City Department of Health and Mental Hygiene, schamberlin@health.nyc.gov
Abstract: The HIV spectrum of engagement in care, or "care cascade," depicts the success of HIV-infected persons in meeting successive stages of the care continuum: linkage to care, retention in care, antiretroviral medication use, and viral suppression. The New York City (NYC) Department of Health and Mental Hygiene (DOHMH) oversees the federal Housing Opportunities for Persons with HIV/AIDS (HOPWA) grant in NYC, which addresses housing needs of low-income HIV-infected persons. We merged HOPWA program data with NYC HIV surveillance data, to create a care cascade restricted to clients of DOHMH-administered HOPWA programs. In addition to comparing the HOPWA cascade against national and citywide standards, we specifically examined clients who "fell off" at each stage of the care cascade. We observed significant sociodemographic differences comparing these clients to those successfully completing the cascade. These findings will assist NYC policymakers in designing program models that reinforce adherence to the HIV care continuum.
Poster 237: The Impact Academy: Developing a New Evaluation Training and Technical Assistance Sequence
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Sara McGirr, Michigan State University, mcgirrsa@msu.edu
Jennifer Mortensen, Michigan State University, morten19@msu.edu
Pennie Foster-Fishman, Michigan State University, fosterfi@msu.edu
Karin Tice, Formative Evaluation Research Associates, ktice@feraonline.com
Alissa Parks, AKP Consulting LLC, alissa@akpconsulting.com
Erin Watson, Michigan State University, droegeer@msu.edu
Abstract: This poster describes the "Impact Academy", an evaluation training and technical assistance sequence developed to support the work of the grantees of four different partnering foundations. The sequence was developed in response to evaluation findings showing capacity needs among these grantees in using a systems lens to understand community conditions, developing strategies that target the root causes of community problems, and using data to inform learning and innovation. The Impact Academy is a cohort-based, real-time-use focused training and technical assistance sequence that builds grantees' capacity using an adult learning approach and ongoing coaching and support. Recognizing that many other evaluators have likely encountered similar capacity needs in their work, this poster aims to share the process our team used to develop the Academy, describe the final content and approach used within the training, and provide lessons learned and recommendations for evaluators to develop similar training models within their own communities.
Poster 15: Making the Development of Performance Indicators a Transformational Experience
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Martha McGuire, Cathexis Consulting Inc, martha@cathexisconsulting.ca
Rochelle Zorzi, Cathexis Consulting Inc, rochelle@cathexisconsulting.ca
Abstract: The Accessibility for Ontarians with Disabilities Act, 2005 requires that adoption of the standards for accessibility be monitored and evaluated. They are interested in the extent to which the standards have impacted Ontarians with disabilities, older adults and their families? To understand the impact, the Accessibility Directorate of Ontario (ADO) required a set of key indicators looking at impact on people with disabilites. This presentation focuses on the methods used to develop indicators that were acceptable to the Accessibility Directorate of Ontario as well as to people with disabilities. It will provide methods and tools that supported giving people with disabilities a strong voice while at the same time meeting the needs of government. We will start with an overview of the purpose of the project, present an overview of the methods used (including how accessibility was assured) and provide details about the use of priority sort in this process.
Poster 90: Use of a Baseline Data Analysis to Identify Educational Impact, Practice Gaps and Challenges From Physician and Medical Professional Learners
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Acacia McKenna, American Society of Clinical Oncology, acacia.mckenna@asco.org
Christina Vidal, American Society of Clinical Oncology, christina.vidal@asco.org
Anne Grupe, American Society of Clinical Oncology, anne.grupe@asco.org
Sean Malone, American Society of Clinical Oncology, sean.malone@asco.org
Carey Beth Catalino, American Society of Clinical Oncology, careybeth.catalino@asco.org
Abstract: The American Society of Clinical Oncology's (ASCO) Continuing Medical Education (CME) program provides a comprehensive portfolio of education that is tailored to a multidisciplinary oncology professional audience. Through the development of the 2013 CME Strategic Plan, the CME Subcommittee and staff identified the need to determine practice gaps, educational needs, and learning outcomes at a global level, independent of individual activities. This poster will focus on a baseline data analysis acquired from ASCO's CME program spanning 2010-2012 and comprised of educational impacts, practice gaps and implementation challenges identified from over 231 activities and approximately 82,000 physician and medical professional learners. Information from these analyses was used to gauge whether educational objectives from initiatives were achieved, identify areas of further educational need and served as a foundation for the Strategic Priorities that the CME Subcommittee and supporting staff intend to pursue in the current CME reaccreditation term.
Poster 23: Building Evaluation Capacity: Where to Start and Assessing Where You Are
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Acacia McKenna, American Society of Clinical Oncology, acacia.mckenna@asco.org
Christina Vidal, American Society of Clinical Oncology, christina.vidal@asco.org
Abstract: Non profit organizations are delivering more public services, spending more funds to support programs, and relying more on government funds to support initiatives. In align with these trends, accountability demands are increasing from funders which require more documentation for contractual agreements, use of funds and a more thorough investigation on the impact of work that is being done. The American Society of Clinical Oncology (ASCO) provides lifelong learning for oncology professionals; cancer research; an improved environment for oncology practice; access to quality cancer care; a global network of oncology expertise; and educated and informed cancer patients. The organization is engaged in addressing the demands of the environment and committed to advancing the field of oncology. This panel discussion shares practical approaches toward building evaluation capacity, highlights successes in establishing evaluation capacity at ASCO and shares tools for continuous improvement.
Poster 253: Assess the State of Your Program: Using Theory of Change to Guide Program Evaluation of a Teachers Institute
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Adriana Medina, University of North Carolina at Charlotte, amedina1@uncc.edu
Lakia Scott, University of North Carolina at Charlotte, lscott33@student.uncc.edu
Scott  Gartlan, University of North Carolina at Charlotte, scott.gartlan@uncc.edu
Abstract: The underlying belief regarding teacher institutes is that they will improve student performance by improving teacher quality. How can that process be captured? The purpose of our poster session is to illustrate how we have used a logic model to evaluate different components of a teachers institute in an attempt to capture the process.
Poster 175: An Evaluator's Perspective: A Critique of the Design and Implementation Processes of Pilot Programs
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Christine Meyer, University of North Carolina at Greensboro, ccmeyer@uncg.edu
Aundrea Carter, University of North Carolina at Greensboro, adcarte2@uncg.edu
Emma Sunnassee, University of North Carolina at Greensboro, emsunnas@uncg.edu
Holly Downs, University of North Carolina at Greensboro, hadowns@uncg.edu
Abstract: This poster provides insight into the design, development, and implementation processes of three Science, Technology, Engineering, and Mathematics (STEM) pilot programs by critiquing the evaluation activities, recommendations made by evaluators, and use of the results. Using the initial evaluation reports from the programs, we extract characteristics of the programs (including program design, purpose, and structure; program logistics; sample characteristics; contextual information; evaluation planning and results; etc.) and the interim evaluator recommendations. Additionally, interviews with the program designers to determine their decision making processes throughout the evaluation will be addressed along with personal reflections from the three evaluators involved in the programs. The reflections are supported with evidence gathered from involvement in all aspects of the evaluation including attending meetings, planning the evaluations, collecting data, writing evaluation reports, and providing recommendations. Lastly, we examine the changes that occurred after the implementation of the recommendations and compare relevant information from all programs.
Poster 96: Translating the Evidence Base to Best Practices that Reach Diverse Populations: Lessons Learned in Evaluation of a Tribal Clinic's In-home Asthma Intervention
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Megan Mikkelsen, Washington State Department of Health, megan.mikkelsen@doh.wa.gov
Julia Dilley, Program Design & Evaluation Services, julia.dilley@gmail.com
Abstract: Public health programs often focus on implementing evidence-based practices, yet there is often little documentation of how those practices are implemented, especially to reach diverse populations. The Washington State Department of Health (DOH), funded by CDC's National Asthma Control Program, partnered with one American Indian Tribe to pilot an in-home asthma assessment and intervention through the tribal clinic. The clinic team successfully improved their clinical capacity for asthma classification, Community Health Representatives (CHRs) engaged community members for in-home visits, and patients improved asthma control practices and reduced asthma triggers in the home. The clinic team was generally satisfied with the improved protocols for supporting patients with asthma, and identified clear recommendations for implementation in other tribes. The DOH goal of assuring the feasibility of this Best Practice in a tribal setting was met, but numbers of patients reached with interventions were less than projected.
Poster 89: Data Quality and Process Evaluation Issues of a Rare Condition Registry: Emerging Lessons From the National Spina Bifida Patient Registry
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Angela Montesanti, Centers for Disease Control and Prevention, www9@cdc.gov
Julie Bolen, Centers for Disease Control and Prevention, jcr2@cdc.gov
Judy Thibadeau, Centers for Disease Control and Prevention, csn2@cdc.gov
Elisebeth Ward, Centers for Disease Control and Prevention, ikt3@cdc.gov
Abstract: Clinic-based registries are commonly used to collect information on rare diseases. However, no standards exist for rare disease registries (AHRQ, Registries for Evaluating Patient Outcomes: User's Guide, 2012), therefore evaluations of their effectiveness are limited. The purpose of this project is to evaluate recruitment, case ascertainment and data collection processes in an initial 10-clinic cohort of the National Spina Bifida Patient Registry (NSBPR). We focus on three questions related to problems that can ultimately affect data quality and describe the mechanisms deployed by the clinics to address these problems. Preliminary findings indicate lack of uniformity among clinics in processes of recruitment, case ascertainment and data collection. Final results will enable us to identify implementation methods and describe how they are being addressed to improve NSBPR data quality, eventually contributing to development of registry standards and additional evaluation criteria to identify and resolve issues affecting data quality in rare disease registries.
Poster 101: Educational leaders' Practices Surrounding the Interpretation and Use of Data in K-12 settings
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Bianca Montrosse Moorhead, Western Carolina University, bemontrosse@wcu.edu
Meagan Karvonen, University of Kansas, karvonen@ku.edu
Abstract: This paper presents the results of the first phase of a qualitative study that investigated current educational leaders' practices surrounding the interpretation and use of data in K-12 settings. Purposeful and reputational case sampling techniques were used to identify and recruit K-12 leaders in North Carolina who had reputations for using evidence to ground decision-making. The primary data collection strategy was in-depth interviews that focused on topics including (1) current practices, (2) personal strengths and areas for growth in data use, and (3) strengths and areas for growth in data use among key stakeholder groups (e.g., school improvement teams). Preliminary data suggests similarities and differences in how K-12 leaders think about and use evidence to ground decision-making. Results will inform future redesign of graduate level evaluation courses.
Poster 255: Measuring a Health-Promotion Resource when Actual Health Behaviors Can't Be Measured
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Heather Moore, Be The Match, hmoore@nmdp.org
Ellen Denzen, Be The Match, edenzen@nmdp.org
Jaime  Preussler, Be The Match, jpreussler@nmdp.org
Abstract: It is estimated that there are currently 100,000 marrow and cord blood transplant (transplant) survivors in the US. Despite improved outcomes, recipients are often lost to systematic follow-up due to the shift from specialists to local physicians in providing survivorship care. As a result, the quality of survivorship care is often suboptimal. Further, patients rarely discuss information they find important with their provider. To promote optimal utilization of recommended screening and preventive practices and facilitate patient-provider communication, survivorship care guides were developed for recipients. This paper will describe a theory-based evaluation methodology for measuring effectiveness of a health-promotion resource for survivorship care, including using a repeat-measures design and the Tailored Design Method for survey administration. Demographics will be used to characterize target audiences and describe the representativeness of evaluation findings. Implicatio ns include best practices for measuring patient-provider information sharing and health-promoting behaviors.
Poster 222: Understanding Leadership From a Youth Perspective
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Jennifer Mortensen, Michigan State University, morten19@msu.edu
Pennie Foster-Fishman, Michigan State University, fosterfi@msu.edu
Kareemah Abdullah, Community Anti-Drug Coalitions of America, kabdullah@cadca.org
Abstract: Though decades of research exist on leadership and our evolving ideas of what makes a leader, few studies address how youth define leadership. Much of the adult leadership literature is based in business/organizational settings, and often this literature is used to inform youth leadership development efforts. It is critical to understand how youth define leadership in their lives so we can create programs that respond to their ideas and meet their needs. This paper will discuss a project that used concept mapping with a group of youth to generate ideas about leadership as well as findings from a survey that explored these ideas with a sample of youth from across the country who were attending a leadership conference. This paper will present the youth-generated ideas about leadership as well as recommendations for how to support youth as leaders within their communities.
Poster 188: Linkage to Care for HIV/AIDS: Service Use and the Effects of Stigma, Mental Health and Substance Use
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Snigdha Mukherjee, Louisiana Public Health Institute, smukherjee@lphi.org
Sarah Chrestman, Louisiana Public Health Institute, schrestman@lphi.org
Russell Brewer, Louisiana Public Health Institute, rbrewer@lphi.org
Megan Wright, Louisiana Office of Public Health, megan.wright@la.gov
Thomas Carton, Louisiana Public Health Institute, tcarton@lphi.org
DeAnn Gruber, Louisiana Office of Public Health, deann.gruber@la.gov
Abstract: It is well known that stigma of HIV, substance use and mental health issues have the potential to affect patients by delaying treatment seeking or even drop out of treatment. Louisiana Positive Charge program focuses on linkage to care of newly diagnosed HIV clients and those out of care but living with HIV infection. Data indicates that a health system and patient-centered approach in Louisiana has facilitated the successful linkage of individuals living with HIV infection into care. At baseline, 11% had an undetectable viral load and a mean CD4 cell count of 293 and at six-month follow-up 22% of participants had an undetectable viral load and mean CD4 cell count of 348. However, participant assessment reveals that on average many participants knew they were HIV seropositive for over 4 years and indicated that some of the biggest barriers to care was stigma, substance use and mental health issues.
Poster 128: Effectiveness of On-Farm Food Safety Workshops: Implications for Program Improvement
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Roshan Nayak, Penn State University, rkn112@psu.edu
Daniel Tobin, Penn State University, dbt127@psu.edu
Rama Radhakrishna, Penn State University, brr100@psu.edu
Joan Thomson, Penn State University, jst@psu.edu
Luke Laborde, Penn State University, lfl5@psu.edu
Abstract: Each year foodborne illness affects approximately 48 million people and takes the lives of thousands in the United States (Center for Disease Control and Prevention, 2012). Microbial contamination of farm produce has been identified as a source for foodborne disease outbreaks and contamination of produce (Sivapalasingam, Friedman, Cohen, & Tauxe, 2004). To prevent this, the U.S. Food and Drug Administration (FDA) have proposed food safety rules for produce growers. These proposed rules cover workers health and hygiene, quality of agricultural water, the use of domesticated animals, potential contamination by wild animals, sanitation standards for equipment, tools and buildings, and traceability/recall. These growers are increasingly required to provide evidence that they are following and can document Good Agricultural Practices (GAPs). This presentation focuses on the outcomes of GAP certificate workshops organized to provide participating growers the knowledge and skills they need to comply with and verify GAPs on their farms.
Poster 226: Measuring Collaboration Among Grant Partners: A Mixed-Methods Approach
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Sandra Newton, University of Minnesota, newto122@umn.edu
Eileen Harwood, University of Minnesota, harwo002@umn.edu
Diane Kampa, University of Minnesota, dkampa@umn.edu
Bruce Alexander, University of Minnesota, balex@umn.edu
Abstract: Collaboration is foundational in large-scale grant program activities, and increasingly funders expect evidence of its outcomes. Measures of collaboration primarily are limited to assessing the strength of partnerships. In this research, I supplemented Frey et al.'s, (2006) Stages of Collaboration quantitative instrument with a semi-structured qualitative interview. Participants are five groups of researchers comprising the interdisciplinary grant partners (funded by the Upper Midwest Agricultural Safety and Health Center). Participants identified current collaborative partners; assigned each partnership a descriptive level indicator; discussed formation of the collaboration; identified their desired collaboration levels by year four of their grant; and discussed how they foresee achieving desired collaboration levels. Preliminary findings suggest this method elicited participants' rich descriptions of their relationships, thus adding a novel dimension to measuring collaboration. This approach appears to facilitate action-oriented thinking (e.g., identifying obstacles to collaboration and strategies to leverage partnerships). The relationships described by participants are visually mapped.
Poster 123: Telling the Story of At-Risk Youth: Utilizing Mixed Methods in Program Evaluation
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Emily Nguyen, University of Nebraska at Omaha, eknezetic@unomaha.edu
Abstract: This poster will focus on sharing the methodologies of utilizing mixed methods in evaluating youth's experiences in a program focused on ensuring at-risk youth are career-ready. Data was collected from a satisfaction survey and focus groups with at-risk youth. This poster will highlight the methodologies used to conduct and analyze this research as well as the learning experiences from conducting community-based research.
Poster 228: Drawing a Representative Household Survey in Burkina Faso Using Multi-Stage Cluster Sampling
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Linh Nguyen, Saarland University, contact.linh.nguyen@gmail.com
Abstract: When evaluating interventions in developing countries, it is difficult to draw a representative sample through probability sampling. Since programmes in international developments often target large populations like the inhabitants of a city, region or even a country, household surveys estimating real population parameters are crucial to the of those programmes. However, there are problems of the sampling frame and coverage errors. With a case study of a household survey on energy-efficient cooking stoves in Burkina Faso, this poster will show how to handle those issues by relying on the national method of enumeration for taking a census. The objective is to demonstrate the merits of a multi-stage cluster sampling going through the sampling process and stages step by step. As such, it shows the importance of refined probability sampling for large scale surveys in the context of evaluations. Finally, the poster will also included unforeseeable challenges encountered in the field.
Poster 70: Civil Society and the Challenges of Governance Evaluations in Zambia
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
John Njovu, Zambia Revenue Authority, njovuj@yahoo.com
Abstract: In Zambia,we have started the process of localising governance assessments. The government has partnered with civil society in carrying out governance monitoring and evaluation. This is done through the Governance Monitoring and Evaluation Committee (GMEC). In 2008, GMEC with the help of GTZ and Inwent Germany undertook to prepare the indicators for monitoring and evaluating governance in Zambia. A study/evaluation of governance in Zambia was carried out by the government's Central Statistical Office. The report was released in 2012 after a change of government. GMEC has also carried out evaluations of some of the governance programmes in the Fifth National Development Plan (2006 - 2010). It has also been instrumental in the preparation of the governance chapter of the Sixth National Development Plan. I critically review the process of these evaluations and the lessons learnt.
Poster 195: Evaluating Community Level Supply Chains in Sub-Saharan Africa Using a Theory of Change Model
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Megan Noel, John Snow Inc, mnoel@jsi.com
Sarah Andersson, John Snow Inc, sandersson@jsi.com
Michelle Pahl, John Snow Inc, mpahl@jsi.com
Yasmin Chandani, John Snow Inc, ychandani@jsi.com
Abstract: The Improving Supply Chains for Community Case Management of Pneumonia and Other Common Diseases of Childhood (SC4CCM) project designed a theory of change (TOC) model that clearly maps the project's goals and objectives and the steps it will take to achieve them. SC4CCM aims to demonstrate that supply chain constraints at the community level can be overcome through interventions in three sub-Saharan African countries. The SC4CCM TOC model identifies the distinct building blocks that, in combination, comprise an effective supply chain for reaching community health workers in remote areas. The TOC links interventions, assumptions and indicators to final outcomes and provides a roadmap for measuring and institutionalizing change. While the building blocks are relevant for all supply chains for CCM medicines, the individual interventions and targets for each indicator tested during the project's lifetime are country-specific. The TOC model was used to develop country-specific implementation and evaluation plans.
Poster 197: Measuring Institutional Performance Improvement in Five West Bank Ministries
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Peggy Ochandarena, Chemonics International Inc, pochandarena@chemonics.com
Garrett Dorer, Chemonics International Inc, gdorer@chemonics.com
Abstract: Five Palestinian Authority ministries in the West Bank received assistance from USAID to improve their delivery of services to citizens and enhance their institutional functioning. Baseline data at project inception was taken as well as at the project's close. A framework for organization development was tailored to reflect seven core areas (with 36 subfactors) for government institutions: governance, administration, human resources, financial resources, organizational management, program management, and service delivery. Explicit scoring criteria was used to rate the ministries on each subfactor and devise a numerical score. Differences in scores were compared in all areas, where the project assisted or did not. Lessons learned about assistance for institutional development are drawn, based on quantitative and qualitative data.
Poster 18: Where are They now? Using Technology to Track Undergraduate Student Researchers: Perspectives From Three Rural States
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Sharon O'Connor, Dartmouth College, sharon.oconnor@dartmouth.edu
Jennifer Harris Forrester, University of Wyoming, jforres5@uwyo.edu
Sarah Olimb, Essentia Institute of Rural Health, sarahtolimb@gmail.com
Pat Conway, Essentia Institute of Rural Health, pat.conway@eirh.org
Virginia Reed, Dartmouth College, virginia.reed@dartmouth.edu
Abstract: The Institutional Development Award (IDeA) program leverages the competitiveness of academic institutions traditionally underrepresented in National Institute of Health (NIH) grants. IDeA Networks of Biomedical Research Excellence (INBRE) programs provide access to biomedical research training for undergraduate students who would otherwise rarely be exposed to similar opportunities. Numerous challenges exist in the evaluation of INBRE's impact on these undergraduate students, especially in rural states where institutions are spread far and wide. This challenge is increasingly evident after students graduate: how do we ensure continued participation in our evaluation efforts? Presentations by INBRE evaluators in Wyoming, North Dakota, and New Hampshire describe theory and activities associated with tracking undergraduate students in this era of increased awareness of email hacking, identity theft, and need for digital security.
Poster 87: Evaluating Utah High School Educators Level of Professional Preparation in Working With LGBTQ Students
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Camille Odell, Utah State University, camille.odell@usu.edu
Rachel Peterson, Utah State University, rachel.peterson@aggiemail.usu.edu
Abstract: LGBTQ youth are at higher risk for suicide, homelessness, dropout and other negative outcomes. Attending a school with a safe and welcoming environment and supportive adults can prevent these outcomes and assist in positive development. To create a safe and supportive school environment, educators must be trained to work with LGBTQ youth, including recognizing and intervening with derogatory and potentially threatening interactions between students in the school. This study was designed to evaluate the degree to which Utah high school educators felt they were prepared to create a safe and welcoming school climate and advocate for LGBTQ high school students. A survey was sent to all administrators, school counselors and teachers in Utah public high schools to assess the amount of training they had received in working with LGBTQ youth, and their levels of comfort and confidence in providing support to LGBTQ students.
Poster 154: Evaluability Assessment: Stakeholders' Workshops for iDentifying the Political-organizational Context in the Implementation of Health Care Networks in Brazil
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Egléubia Oliveira, Fiocruz Foundation, biaol@ensp.fiocruz.br
Gisela Cardoso, National School of Public Health, gisela.cardoso@gmail.com
Marly Cruz, Fiocruz Foundation, marly@ensp.fiocruz.br
Ana Cristina Reis, Fiocruz Foundation, anareis@ensp.fiocruz.br
Angela Casanova, Fiocruz Foundation, angelacasanova@ensp.fiocruz.br
Andre Guerrero, Fiocruz Foundation, andreguerrero@fiocruz.br
Sueli Rodrigues, Brazil Ministry of Health, sueli.moreira@saude.gov.br
Abstract: As part of the evaluability assessment of Brazilian's Ministry of Health strategic project "Training and Quality Improvement of Health Care Network", a political and organizational analysis was conducted. This project aims to support the implementation of Health Care Networks (RAS), based on five health issues of primary care, in 15 Brazilian regions. Fifteen workshops were held between December 2012 and March 2013, with stakeholders from the 15 regions. Analysis took into account strengths and weaknesses listed by local regional supporters, and discussed with the group of stakeholders. Major weaknesses were: health management centralization at state regional level and low participation of the municipalities involved, as well as the happening of local elections. Major strengths were: alignment between the project's propositions and state health plan, and commitment of most local supporters. A strategic plan was elaborated in each of the 15 workshops as to address encountered barriers and accelerate implementation process.
Poster 142: Partnerships in a Multi-Level, Multi-Site Evaluation: the Health Foundation of South Florida's Healthy Eating Active Communities (HEAC) Initiative
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
James Pann, Nova Southeastern University, pann@nova.edu
Angela Yehl, Nova Southeastern University, yehl@nova.edu
Elda Veloso, Nova Southeastern University, veloso@nova.edu
Abstract: The obesity epidemic continues to pose a major public health concern (NIH, 1998) and in early 2007 the Health Foundation of South Florida (HFSF) embarked on a five-year responsive grant-making initiative, "Healthy Eating Active Communities" (HEAC). The project adopted a multi-sector orientation and expanded its partners beyond traditional health-related organizations. An evaluation team was contracted to conduct the HEAC program evaluation and collaborated with the HFSF to develop the project logic model, guide the development of the evaluation plan, and evaluate the outcomes associated with each of the grantees' programs. In this way, the relationships between the funder, grantees, and evaluator demonstrate a uniquely beneficial partnership in implementing and evaluating large multi-County, multi-sector obesity prevention programs. The intention of this presentation is to provide an example of the potential for partnerships to facilitate effective program implementation in this area, utilize practical program evaluation, and promote program growth and sustainability.
Poster 156: Use of Secondary Data in Extension Program Development
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Dhruti Patel, University of Maryland, dhrutip@umd.edu
Teresa McCoy, University of Maryland, tmccoy1@umd.edu
Patsy Ezell, University of Maryland, pezell@umd.edu
Abstract: There are various advantages to using secondary data to drive program development. This poster focuses on an evaluation project to analyze secondary data collected by the Maryland Department of Aging (MDA). Data was shared for: 1) MDA to understand eating preferences, chronic health conditions, and quality of the meal program provided to senior participants 60 years or older, and 2) UME to understand the demographics and preferences of the senior population for better nutrition education programs. There were 2,247 senior participants who took part in the survey. The data indicated that seniors have a high preference (79%) for fresh fruits and low preference for legumes (23%) and ethnic foods (27%). The data also suggested that there is a higher prevalence of arthritis (49%) and hypertension (39%) than other health conditions. Plans are underway for scientific studies to delve into these preliminary findings using rigorous methods that can allow for generalizability.
Poster 57: The Art of Evaluation Study Support
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Beth Peery, Magnolia Consulting LLC, beth@magnoliaconsulting.org
deKoven Pelton, Magnolia Consulting LLC, dekoven@magnoliaconsulting.org
Abstract: Successful evaluations are multifaceted undertakings built from the ground up. Two lower rung components that lead to an effective evaluation are program coordination and data management. Systematic practices in these areas factor into consistently high quality data and lead to reliable evaluation analyses and outcomes. Yet, the specific details of strong project support and the development of a comprehensive approach to data management are rarely discussed in depth and are frequently overshadowed in the big picture of evaluation. In an effort to share knowledge and experience with the evaluation community, this poster presents the views of two professional support staff on their procedures and methodologies developed to ensure that small and large evaluation projects alike run smoothly and provide accurate data for analysis. It highlights challenges and provides strategies to meet them, and offers best practices, tricks of the trade, and vignettes in order to open a dialogue among evaluators.
Poster 95: Evaluating a Treatment Program for Cultural Change: Key Challenges and Opportunities Facing Nonprofit Evaluators in the Mental Health Services Field
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Althea Pestine, Texas Network of Youth Services, arpestine@utexas.edu
Abstract: This poster draws from the perspective of the evaluation of a program implementing the Substance Abuse and Mental Health Services Administration's (SAMHSA's) Six Core Strategies to Prevent Conflict and Violence to help Texas residential treatment centers (RTCs) reduce the need for seclusion and restraint. The program aims to create a "Culture of Care" by utilizing a curriculum of client-centered, strengths-based mechanisms to minimize the need for physical intervention. This analysis provides a framework for adapting the evaluation of such a project to the specific issues that arise from the sensitivity of relationships required to enact a program of cultural change.
Poster 171: A Formative Evaluation of an Ethnomathematics Educational Program
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Joanna Philippoff, University of Hawaii at Manoa, philippo@hawaii.edu
Lisa Vallin, University of Hawaii at Manoa, vallin@hawaii.edu
Linda Furuto, University of Hawaii at West Oahu, lfuruto@hawaii.edu
Abstract: Ethnomathematics-the study of the relationship between mathematics and culture-programs can be difficult to evaluate because of untraditional course delivery, diverse participant backgrounds, and challenges to evaluating difficult-to-define objectives, such as the development of a supportive community. We present findings from a formative evaluation of an ethnomathematics summer institute, a two-week course including weekends, for undergraduates at the University of Hawaii-West Oahu, designed to address issues of retention, support, and advancement in STEM majors, particularly for Native Hawaiian students. We used a mixed-methods approach, including observations, surveys, interviews, authentic assessments, and student tracking to examine the association between program participation and student outcomes. In addition to presenting the study's findings, we will elaborate on the evaluation process, sharing the challenges and successes of evaluating a transdiscipline program that links history and culture to 21st century problems and skills.
Poster 82: Using Longitudinal Randomized Control Trials in Schools: Challenges and Solutions From a Five-State Study
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Ashley Philliber, Philliber Research Associates, aphilliber@philliberresearch.com
Sally Brown, Philliber Research Associates, sallypra@compuserve.com
Susan Philliber, Philliber Research Associates, sphilliber@philliberresearch.com
Abstract: While a longitudinal, randomized control trial has long been thought of as the "gold standard" in evaluation, the use of this design in a real world context can often be difficult. This paper discusses the challenges in using this design across five states to test the efficacy of a teen pregnancy prevention program in 104 groups. In schools, issues arise from the time of obtaining consent to ensuring high follow-up rates. Substantial educational efforts are needed as program and school personnel may have little knowledge about this design and little motivation to see the study completed with integrity. This paper reports the explanations provided about choosing school classes, how randomization was accomplished, incentives used for school administrators, teachers, and students, mid-year schedule changes that threatened the design, techniques for follow-up at one- and two-year intervals, and strategies used to maintain a high survey completion rate while maintaining the original sample.
Poster 61: Using Patron Surveys and Mapping Software to Evaluate a Food Bank
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Russell Phillips, California State University at Northridge, russell.phillips.55@my.csun.edu
Jodi Brown, California State University at Northridge, jodi.brown@csun.edu
Abstract: Current economic conditions have forced people into financial hardship, resulting in greater need for effective assistance provided by food pantries. This evaluation uses patron surveys and geographic mapping to analyze the current food distribution system of a Southern California food pantry to determine whether basic food needs are being met. Anonymous surveys (n=87) were distributed to patrons of the food bank. Respondents were asked to comment upon the variety of food received, amount of time food lasted, safety of environment, and major cross streets from which they traveled to get to the food pantry. ArcGIS mapping software was used to define the reach of the local food bank. Incorporating 2010 census data in the map(s) resulted in clear snapshots of how the food bank is serving the local community. Providing a visual representation of service delivery gives stakeholders a clear indication of potential gaps in service coverage or over coverage.
Poster 251: Making Use of Fidelity Output Data and Implementation Evaluation Data to Avoid Type III Error, Better Understanding of Facilitator Instruction, Accountability to Funding Agency, Program Improvement and Covariates in Analytic Model
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Harry Piotrowski, ITMESA LLC, zhp@sprynet.com
Abstract: Background: Without measurement adherence to the intended instructional model, there is no way to determine whether unsuccessful outcomes reflect failure of the model or failure to implement the model as intended. Objectives: Multiple methods to assess fidelity (output) and implementation evaluation will illustrate different types of data. Settings: Northwest Indiana urban (African American, Hispanic, 7th graders) and rural (White, 6th, 7th graders) communities, school-based intervention. Analysis: Descriptive statistics with univariate (chi-square, t-tests) and multivariate analyses. Criterion and norm-referenced standards are discussed for fidelity and implementation indicators. Results: Multi-methods examples illustrate: Content (what is being taught); Pedagogy (how the content is taught); Implementation (logistics conducive to optimal learning environment); Fidelity Monitoring (adherence to core curriculum elements). Measurement tools will include: Participant Attendance Log, Program Activities Matrix, Educator Self Assessments, Independent Observer Assessments, Student/Participant .Assessment, Teacher Assessment, Participant Interviews, Focus Group, Participant Retrospective Assessment. Conclusion: Results demonstrate the value of variability (SD and skewness) at several levels and criterion-referenced data across time (drift), facilitator pairs, schools, and classroom) to assure completeness and fidelity.
Poster 219: Cross-cultural Translation: Evaluating End-user Perspectives on Implementing an Evidence-based Algorithm for Screening and Diagnosing Tuberculosis Among People Living With HIV in Kenya, Thailand, and Vietnam
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Lauren Polansky, Centers for Disease Control and Prevention, lauren.s.polansky@gmail.com
Eric Pevzner, Centers for Disease Control and Prevention, ecp9@cdc.gov
Le Hung Thai, Centers for Disease Control and Prevention, leth@vn.cdc.gov
Sara Whitehead, Centers for Disease Control and Prevention, svw7@th.cdc.gov
Thuy Trinh, Centers for Disease Control and Prevention, trinhtt@vn.cdc.gov
Benard Ochuka, Centers for Disease Control and Prevention, benochuka@yahoo.co
Tareerat Chemnasiri, Centers for Disease Control and Prevention, tareeratc@th.cdc.gov
Wanitchaya Kittikraisak, Centers for Disease Control and Prevention, glr9@cdc.gov
Sean Cavanaugh, Centers for Disease Control and Prevention, hgi7@cdc.gov
Surbhi Modi, Centers for Disease Control and Prevention, bkt1@cdc.gov
Barbara Burmen, Centers for Disease Control and Prevention, bburmen@kemricdc.or
Abstract: Translating evidence-based innovation into practice is complex. In cross-cultural environments, diverse factors can influence understanding, adoption, and implementation. Formative evaluation with local staff responsible for implementation (end-users) can help elucidate context-specific factors that may promote or inhibit translation. We used qualitative methods to 1)explore why an evidence-based approach to TB screening and diagnosis in people living with HIV may or may not be adopted as intended, and 2) understand what could be done to improve adoption from the perspective of the end-user. We describe how we incorporated attributes of the diffusion of innovation theory into focus group discussions to capture end-user perspectives across three different operational contexts: Kenya, Thailand, and Vietnam (Rogers, 1973). We present the common and unique factors of adoption, and explore the benefits and challenges of a horizontal approach to data analysis.
Poster 69: Assessing Science, Technology, Engineering, and Mathematics (STEM) Education Research Connections
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Alan Porter, Georgia Institute of Technology, alan.porter@isye.gatech.edu
David Schoeneck, Search Technology Inc, daves@searchtech.com
Gregg Solomon, National Science Foundation, gesolomo@nsf.gov
Heena Lakhani, National Science Foundation, hlakhani@nsf.gov
James Dietz, National Science Foundation, jdietz@nsf.gov
John O'Brien, Georgia Institute of Technology, jobrien9@gatech.edu
Abstract: We analyze Science, Technology, Engineering & Mathematics ("STEM") education's multi-disciplinary components, and the "connections" (knowledge interchange) among them and with related fields. Education communities increasingly draw upon findings, models, and methods from other fields and areas of expertise. Researchers and practitioners from many disciplines are engaging in STEM education. Our combined bibliometric and text analysis techniques measure and map cross-disciplinary knowledge interchange. We assess 166 projects funded by NSF's Research and Evaluation on Education in Science & Engineering ("REESE") program. Herein, we key on some 1000 research outputs attributed to REESE support to classify them by research area and to study their research community intersections, based on co-authoring and cross-citation links. We address five prime research questions relating to knowledge interchange within and between STEM Education, cognitive sciences, and other STEM disciplines.
Poster 129: Instruments to Predict Research to Practice Relevance
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Louise Quijano, Colorado State University, louise.quijano@colostate.edu
Victoria Buchan, Colorado State University, victoria.buchan@colostate.edu
Abstract: Program evaluation utilizes social research methods "to systematically investigate the effectiveness of social intervention programs in ways that are adapted to their political and organizational environments and are designed to inform social action in ways that improve social conditions" (Rossi, Lipsey & Freeman, 2004, p. 29). In the Agriculture, Forestry and Fishing Centers, the role of evaluation addresses these environments on an individual project basis. The High Plains Center Agricultural Health and Safety (HICAHS) is committed to improving worker safety and health through the implementation of a comprehensive system of evaluation that assesses processes and impact of its Core research programs and projects. The RE-AIM Framework (http://www.RE-AIM.org) is used in conjunction to assess implementation of specific projects in the community. To capture Center productivity and to document impact, scaled instruments have been developed to track the "Research-Practice Integration" (r2p) potential of projects in each major Core. (Urban & Trochim,2009).
Poster 6: Evaluability Assessment to Technical Assistance to Outcome Evaluation of Anti-Trafficking projects funded by the United States Department of State, Office to Monitor and Combat Trafficking in Persons (J/TIP)
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Beth Rabinovich, Westat, bethrabinovich@westat.com
Tamara Daley, Westat, tamaradaley@westat.com
Andee Parks, International Justice Mission, aparks@ijm.org
Abstract: This session focuses on how the results of evaluability assessments of J/TIP-funded anti-trafficking projects led to two different paths: (1) delivery of ongoing technical assistance and (2) the conduct of an outcome evaluation. Through J/TIP, the State Department funds anti-trafficking projects around the world. The projects discussed in this session are in South Asia and Latin America. The discussion will address the factors that led to the different evaluation activities, how the evaluators built ongoing relationships with the J/TIP anti-trafficking projects, the collaborative approach to selecting areas for technical assistance, and the approach to conducting the outcome evaluation. There will be an emphasis on how the technical assistance and outcome evaluation utilized existing data collected by the J/TIP projects. The session will conclude with discussions of the usefulness of the evaluation activities from the perspective of the project staff, as well as promising practices in building ongoing relationships with NGOs.
Poster 65: Higher Education, iPads, and Evaluation: A New Paradigm in Collaboration
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Jennifer Reeves, Nova Southeastern University, jennreev@nova.edu
Candace Lacey, Nova Southeastern University, lacey@nova.edu
Glenda Guntner, University of Central Florida, glenda.guntner@ucf
Anymir Orellana, Nova Southeastern University, orellana@nova.edu
Andrea L Buenano, Nova Southeastern University, abuenano@nova.edu
Abstract: As the digital generation enters institutions of higher education, colleges and universities are attempting to attract students by integrating the use of mobile devices into the curriculum with some schools giving every incoming student an iPad. This presentation focuses on the design, implementation, and evaluation of an online Master's program where faculty and students were given iPads to compliment a new curriculum focused on full technology integration. Our team of four professors and a graduate student clearly charted new waters in collaboration and evaluation. We are creating a new model of evaluation that demands that we acknowledge our biases, be responsive to changes posed by the technology, collaborate on a regular basis, share roles and responsibilities, and understand that while we might be building the plane while flying it we are headed in the right direction.
Poster 120: Using Tablet Technology in the Field to Provide Evaluation Data and Improve Practice
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Brad Richardson, The University of Iowa, brad-richardson@uiowa.edu
Janet Horras, Iowa Department of Public Health, janet.horras@idph.state.ia.us
Kellee McCrory, The University of Iowa, kellee-mccrory@uiowa.edu
Abstract: The evaluation of Maternal, Infant, and Early Childhood Home Visiting Program is intended to assist the state and federal governments in efforts to strengthen and improve comprehensive home visiting services. Through evidence-based home visiting programs for pregnant women, expectant fathers, parents and primary caregivers of children birth to kindergarten, improvements in health and prevention of child injuries, reductions in the incidence of child maltreatment and improved school readiness are intended. Data have been collected for two years, while services are being provided in homes, through the use of tablet technology utilizing iPads with wireless access to the university-based web-enabled data collection system called REDCap (Research Data Capture). Data are then immediately available for analysis in a variety of statistical packages (e.g., SPSS, R, SAS, Stata). Results have been routinely provided to stakeholders to aid in program improvement and management as well as for required reporting of performance measures.
Poster 132: Developing Age-specific Measures to Assess Student Knowledge: Evaluating the Brain Power Science Curriculum
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Carina Rivera, University of Nevada, Reno, crivera@casat.org
Stephanie Asteriadis, University of Nevada, Reno, sasteriadis@casat.org
Joyce Hartje, University of Nevada, jhartje@casat.org
Marie Tully, University of Nevada, Reno, mtully@washoeschools.net
Abstract: Drug and alcohol use are high among youth, and those who engage in such high-risk behaviors are more likely to use and have problems with substance use in adulthood. Opportunities to engage students in prevention are limited, especially in school-based settings. To address this, a science curriculum was designed by the National Institute on Drug Abuse to increase knowledge about science, the effects of substances on the brain and body, and how to use critical thinking to make healthy choices, for infusion into core curricula. The curriculum aligns with Nevada and National Education Standards for four different grade levels: kindergarten and first, second and third, fourth and fifth, and sixth through eighth. A separate pre- and post-test was developed for each grade cohort that addressed the varying degrees of literacy and reading comprehension among Nevada students. Attention to developmental factors is necessary to accurately assess intervention outcomes.
Poster 150: Blending Public Health, Evaluation, and Adult Education Theory to Assess Formal and Informal Learning Outcomes in a Public Health Certificate Program: Learning From Kirkpatrick, Scriven, Brookfield, Mezzirow and Our Own Evaluation Practice
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Victoria Rivkina, DePaul University, victoria.rivkina@gmail.com
Leah Neubauer, DePaul University, lneubaue@depaul.edu
Cynthia Tucker, AIDS Foundation of Chicago, ctucker@aidschicago.org
Abstract: This presentation will detail the theory-informed evaluation design process and the multidisciplinary evaluation framework which emerged to assess the learning outcomes of a week-long public health residential training program. The program provides 63 hours of public health training to community-based organization (CBO) HIV prevention and treatment service providers. Across five annual cycles, the evaluation team employed multiple-method rapid formative evaluation, process evaluation, and currently, an outcome evaluation aimed at assessing the short, mid and long-term participant learning outcomes (n=54 participants). Daily evaluation forms focus on content, applicability, format, and structure. Short-term outcomes are assessed quarterly post-training, and the mid/long-term outcome evaluation assesses participant's learning outcomes, enhanced leadership abilities, professional growth, and personal transformation. The presentation will share process and outcome evaluation data. Drawing from five years of evaluation practice, lessons learned from the theory and program-driven approach to evaluating a residential, public health certificate program will be shared.
Poster 74: Using Social Network Techniques to Visualize Structured and Unstructured Activities of Latino Youth
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
James Roebuck, University of Arizona, roebuck@email.arizona.edu
Mari Wilhelm, University of Arizona, mswphd@comcast.net
Abstract: Involvement in structured and unstructured activities has positive effects on academic achievement for Latino youth. In order for school staff to be effective in crafting intervention strategies, it is important to have a holistic picture of the culture of activities that students are involved with. Social Network Analysis (SNA) offers many methods for visualizing relational data. In this poster, we highlight a strategy to visual the involvement data of Latino youth. Using GEARUP data on 10th grade Latino youth we construct a two-mode affiliation matrix of youth by activity. We then visualize this data using Gephi, which is an open source data visualization application. Results of the visualization provide a holistic image of the culture of involvement and highlight "communities of practice" of the Latino youth.
Poster 64: Evaluating the Teen Link Pregnancy Prevention Program: Developing Recruitment and Retention Protocols to Improve Study Outcomes
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Kylie Rowe, University of Nevada, Reno, krowe@casat.org
Jessica Crane, University of Nevada, Reno, jcrane@casat.org
Joyce Hartje, University of Nevada, jhartje@casat.org
Eric Ohlson, Join Together Northern Nevada, eric@jtnn.org
Abstract: Formal protocols used to recruit and retain participants in randomized controlled trials are vital to the integrity of project implementation and evaluation procedures. Particularly, formal protocols can be utilized to navigate unique challenges associated with recruitment and retention of mentors and youth (i.e., BIGs and LITTLEs) from the community-based program, Big Brothers Big Sisters of Northern Nevada (BBBSNN). In early implementation, the Teen Link pregnancy prevention program faced challenges. There was inconsistency among BBBSNN program staff in recruiting parents, BIGs and LITTLEs, and retention was affected by lower than expected participant attendance at Teen Link events. As a result, the Teen Link Recruitment & Retention Protocols Manual was collaboratively designed and implemented through an on-site training with BBBSNN program staff. The protocols are intended to provide a structure to guide program staff, and also serve as a tool that will help quantify the effects of recruitment and retention efforts.
Poster 13: Cost-Effective, Cloud-based Evaluation Tools: Google Sites and Surveys
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Liza Ruzer, Hatchuel Tabernik & Associates, lruzer@htaconsulting.com
Danielle Toussaint, Hatchuel Tabernik & Associates, dtoussaint@htaconsulting.com
Abstract: With the ubiquity of 21st century online data collection and real-time analysis tools, the transfer of information between practitioner, stakeholders, and evaluator is becoming more frequent. Google sites are an inexpensive, easy to use option for practitioners looking to collect and share data with their clients. Additionally, these tools can facilitate a participatory evaluation process - giving parties involved a sense of buy-in and an informed, iterative understanding of their program and findings. This demonstration will walk you through the basics of developing a Google site showing you how easy it is to exchange information in an accessible way with clients and stakeholders involved in the evaluation. We will also discuss the integration of additional Google components such as Drive, Forms, and Calendars into the Google site. As an example, we will share how we built a Google Site for a nutrition education and exercise program in the Bay Area, and how it was received by both program staff and evaluators.
Poster 98: Development of a Methodological Evaluation Tool for Public Health Programs
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Rosina Salerno, Pan American Health Organization, salernor@paho.org
Inmaculada Gisbert Civera, Evaluation Specialist in Pan American Health Organization, inma.gisbert@gmail.com
Blanca Cousio, National Consultant in PAHO Paraguay, cousinb@par.ops-oms.org
Antonieta Arias, National Consultant in PAHO Paraguay, ariasa@par.ops-oms.org
Francoise Barten, Pan American Health Organization, bartenfr@sur.paho.org
Mónica Ruoti, Independent Consultant, monicaruoti@gmail.com
Cristina Arroon, Independent Consultant, crisarrom@gmail.com
Verónica Serafini Geoghegan, Independent Consultant, veronica_serafini@yahoo.com
Stephany Laryea, National Consultant in PAHO Suriname, laryeasteph@sur.paho.org
Elly Van Kanten, National Consultant in PAHO Suriname, vankante@sur.paho.org
Abstract: The quality of design and evaluation of Public Health Programs varies a lot and it needs a methodological tool to identify requirements of quality. PAHO Evaluation Group (PAHO/EV) has been carry out a process to design a methodological tool for Public health managers based on a core of quality items for design and evaluation of Public Health Programs which establish objective minimum requirement for self-evaluation. The initiative was divided into three phases: in the first stage, the EG/PAHO conducted a DELPHI study based on a comprehensive research of literature and with the participation of 45 international experts who contributed to identify and prioritize a list of main elements of quality assurance in Public Health Programs. In the second stage, EG/PAHO carried out a consultation process in selected Latin American countries conducting interviews, surveys and focal groups with the aim of including the perspectives of the actors directly involved in the Programs.
Poster 200: Using Social Network Analysis to Evaluate a National Leadership Network
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Anna Saltzman, Harder+Company Community Research, asaltzman@harderco.com
Clare Nolan, Harder+Company Community Research, cnolan@harderco.com
Meghan Hunt, Harder+Company Community Research, mhunt@harderco.com
Abstract: This poster will present findings and lessons from a social network analysis involving influential stakeholders participating in the US National Oral Health Alliance (Alliance). Founded as a nonprofit organization, the Alliance provides a platform for a diverse network of state and national leaders to forge common ground in order to create viable solutions for improved oral health across the country. Social network analysis (SNA) was used to measure the density and quality of Alliance participant relationships over time, as well as changes in relational patterns among key subgroups with divergent perspectives on oral health programs and policies. The poster will present findings and methodological lessons from the study, which used Tuckman's stages of group development as the basis for its measurement framework.
Poster 233: Process and Outcome Evaluations: Both Necessary in Nutritional Education
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Olga J Santiago Rivera, Michigan State University, santia37@anr.msu.edu
Larissa True, Michigan State University, truelari@msu.edu
Dawn Contreras, Michigan State University, contrer7@anr.msu.edu
Dawn Earnesty, Michigan State University, wilcoxd4@anr.msu.edu
Shayna Danto, Michigan State University, dantosha@msu.edu
Margaret Lashore, Michigan State University, lashore@anr.msu.edu
Beth Jabin, Michigan State University, jabinb@anr.msu.edu
Abstract: A quasi-experimental intervention study (intervention and comparison groups) evaluated the effectiveness of a nutrition education program for seniors (n=707). The objectives were to increase participants' fruit/vegetable consumption and physical activity. To examine the effectiveness of the program, quantitative (outcome evaluation) and qualitative (process evaluation) information were collected. Process evaluation components included adherence to the curriculum, exposure to dose, quality of delivery, participant responsiveness, and program differentiation. Instructors reported high levels of compliance with the nutrition-related components of the curriculum. Outcome results suggest that the program was effective in increasing fruit/vegetable intake among older adults in Michigan. The design and implementation of the evaluation of this program reflect how following the standards of quality in evaluation methods can help meet the objectives of a nutrition education curriculum and explain the results of the outcome evaluation.
Poster 67: "Data Shorts": A Method of Disseminating Public Mental Health System Data in a User-Friendly Format
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Tim Santoni, University of Maryland Baltimore, tsantoni@psych.umaryland.edu
Preston Greene, University of Maryland Baltimore, pgreene@psych.umaryland.edu
Diana Seybolt, University of Maryland Baltimore, dseybolt@psych.umaryland.edu
Susan Bradley, Maryland Department of Health and Mental Hygiene, susan.bradley@maryland.gov
Sharon Ohlhaver, Maryland Department of Health and Mental Hygiene, sharon.ohlhaver@maryland.gov
Stacy Rudin, University of Maryland Baltimore, stacyrudin@verizon.net
Sheba Jeychandran, Maryland Department of Health and Mental Hygiene, 
Qiang Qian, University of Maryland Baltimore, qqian@psych.umaryland.edu
Abstract: "Data Shorts" are brief reports developed by the Maryland Mental Hygiene Administration (MHA) and the University of Maryland, Baltimore Systems Evaluation Center (SEC). They are designed to provide a quick overview of specific topics and analyses of interest to behavioral health system stakeholders. The Data Shorts are disseminated to a variety of state and county administrators as well as mental health service providers and other stakeholders such as the consumers and advocates. Data Short topics are designed to address current topics of importance to the field; recent issues have included analyses regarding mental health recovery, smoking rates for both adult and youth consumers, and general health and wellness. This poster will provide a brief background regarding the development of the reports, share a few examples, and discuss the importance of sharing data with stakeholders using user-friendly methods.
Poster 53: Evaluating the Quality and Effectiveness of the Primary Care Implementation Guide for Alcohol Screening and Brief Intervention (SBI)
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Saloni Sapru, Westat, salonisapru@westat.com
Mary Butler, Westat, marybutler@westat.com
Rachel Gaddes, Westat, rachelgaddes@westat.com
Nancy Cheal, Centers for Disease Control and Prevention, ncheal@cdc.gov
Daniel Hungerford, Centers for Disease Control and Prevention, dhungerford@cdc.gov
Elizabeth Dang, Centers for Disease Control and Prevention, 404-498-3947
Abstract: In the United States 25% of adults drink at excessive levels, placing themselves and others at risk. Medical practitioners often fail to identify risky drinkers. Implementing alcohol SBI universally in primary care will identify many risky drinkers who would otherwise be missed. The Centers for Disease Control and Prevention (CDC) has developed a guide to help medical practices implement and integrate alcohol SBI routinely in clinical services. This evaluation assesses the quality and effectiveness of the guide which is piloted for use with three CDC-funded Fetal Alcohol Spectrum Disorders Regional Training Centers (RTCs) and their associated primary care clinics. The presentation describes how a logic model is developed to understand the context, frame questions, and direct the evaluation. Through onsite interviews with RTC and clinic staff, the evaluation explores the usefulness of this guide in alcohol SBI training, planning, and implementation in diverse clinical practices with multiple stakeholders.
Poster 160: Community-Based Evaluation to Capture Collective Impact
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Laura Scharphorn, HighScope Educational Research Foundation, lscharphorn@highscope.org
Elizabeth Mann, HighScope Educational Research Foundation, emann@highscope.org
Tomoko Wakabayashi, HighScope Educational Research Foundation, twakabayashi@highscope.org
Abstract: This paper presents the Center for Early Education Evaluation at HighScope's (CEEE) community-based evaluation of the Max M. & Marjorie S. Fisher Foundation grantees in Brightmoor, Detroit. Grantees provide services to childcare providers (caregivers) to improve kindergarten readiness. This project leverages evaluation to inform both the continuous improvement of services at the grantee level and progress towards greater collaborative impact at the community level. Specifically, we ask: How do Foundation funds, at the program and community level, impact kindergarten readiness? We test whether caregiver beliefs and practices improve as a function of participation in grantees' programs and, further, whether child outcomes improve with caregiver gains. To assess community-wide impact, CEEE will build a database to measure community-wide treatment dosage for caregivers and evaluate the consequent community-wide child impact. We argue that community-based evaluation can help funders and grantees alike develop collaborative impact analysis that accurately captures progress towards community goals.
Poster 181: Lutheran World Relief's Design, Monitoring, Evaluation and Learning (DMEL) Framework
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Garrett Schiche, Lutheran World Relief, gschiche@lwr.org
Jacque Hlaibi, Lutheran World Relief, jacques@lwr.ne
Abstract: This poster will present Lutheran World Relief's Design Monitoring Evaluation and Learning Framework with the expectation that it will be useful for other development practitioners searching to better integrate M&E into their projects. The framework contains the tools and resources necessary to guide staff in the implementation of each aspect of M&E that takes place throughout the project life cycle. The poster will clearly show how each aspect is linked to the other and how each tool is used to effectively plan, design, implement and use the results from an M&E system. The poster will focus on how the framework was designed not only with content in mind, but also with an emphasis on structure. Content is only as good as the degree of its accessibility and use. The structure was designed to help staff with varied M&E backgrounds access content most relevant to their needs.
Poster 56: Putting Your Money Where Your Monitoring Is! Using 'SMILER' for Monitoring and Evaluation (M&E), Accountability and Learning on a Feed the Future Project in Zambia
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Guy Sharrock, Catholic Relief Services, guy.sharrock@crs.org
Hap Carr, Catholic Relief Services, hap.carr@crs.org
Abstract: Project monitoring has for too long been a poor relation to both project design and project evaluation. Yet for CRS establishing a good project monitoring system helps to improve the impact of development interventions in a numbers of ways: it improves both adaptive project implementation and responsive management; it provides a good foundation for robust project evaluation; and, when done well, it can help to ensure proper accountability to different stakeholders. CRS has been using a system known as SMILER - Simple Measurement of Indicators, and Learning from Evidence-based Reports - with some success over the last few years, and most recently for a Feed the Future program in Zambia. In this last application of SMILER, much emphasis has been given to balancing the needs of donor and project managers for quantitative data with the need for beneficiary accountability so that there is an ongoing process of project learning and adaptation.
Poster 59: Helping Healthcare Teams in Transformation: The Embedded Developmental Evaluator's Role
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Danielle Simpson, Vancouver Coastal Health, danielle.simpson@vch.ca
Laurie Ringaert, Vancouver Coastal Health, laurie.ringaert@vch.ca
Kate Redfern, Vancouver Coastal Health, kate.refern@vch.ca
Carole Gillam, Vancouver Coastal Health, carole.gillam@vch.ca
Carol Park, Vancouver Coastal Health, carol.park@vch.ca
Susan Lim, Vancouver Coastal Health, susan.lim@vch.ca
Abstract: The combination of an embedded as well as a developmental evaluator is a new concept in healthcare transformation. Vancouver Coastal Health is part of the British Columbia Ministry of Health Integrated Primary and Community Care (IPCC) initiative that is working to create transformational changes in health care. The developmental evaluation team has been using a variety of evaluation techniques to assist the project teams to learn and move forward with their improvement projects. A recent provider survey provided feedback to multiple levels of the healthcare team and helped shaped the project in moving forward. Operating under the developmental evaluation principles of utilization focus, empowerment, capacity building, and rapid feedback, this approach has been highly valuable to the management and front-line teams. This presentation explores the relationship between the evaluator and those being evaluated as a vital partnership and essential to the work of healthcare transformation.
Poster 136: Safeguard Policy Implementation in Indonesia: A Monitoring and Evaluation (M&E) perspective
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Ratnayu Sitaresmi, Indonesian Development Evaluation Community (InDEC), rsitaresmi@yahoo.com
Abstract: Following WB definition, Safeguard Policy (SG) objective is to prevent and mitigate undue harm to people and their environment in the development process. ADB's objective in implementing SG is to avoid, or when avoidance is not possible, to minimize and mitigate adverse project impacts on the environment and affected people, and to help borrowers strengthen their safeguard systems and develop the capacity to manage environmental and social risks. As part of effort to protect environment and social life towards impact of development activities, it is important to put SG explicitly into the logical framework of the program or project. This paper will describe some challenges on monitoring, assessing, and measuring the SG implementation, and try to suggest some possibilities indicators for M&E framework development.
Poster 35: Innovative Strategies for Maximizing Recruitment and Retention of Hard-to-Reach Adolescent Research Participants
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Holli Slater, University of Texas at Arlington, holli.slater@mavs.uta.edu
Diane Mitschke, University of Texas at Arlington, 
Abstract: An evaluation plan for pregnancy prevention programs that work with vulnerable adolescents requires innovative strategies in order to maximize enrollment numbers and minimize attrition. Challenges often encountered include difficulty in recruiting participants to the study, obtaining parental consent and adolescent assent, developing a sense of trust between the participants, their parents, and the research team, and locating transient participants for the duration of the study. These difficulties are often compounded when program content relates to adolescent sexual health and pregnancy prevention due to the sensitive nature of topics discussed. This poster presentation will present a number of creative approaches that were developed to address these challenges for the evaluation of Crossroads, an Office of Adolescent Health-funded pregnancy prevention program targeting older adolescents.
Poster 210: Performing Arts and Health Promotion: Using Student-led Participatory Evaluation to Assess Audience Engagement
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Annelise Smith, Washington State University, annelise.smith@email.wsu.edu
Paula M Adams, Washington State University, padams@wsu.edu
Laura Hill, Washington State University, laurahill@wsu.edu
Abstract: A growing number of programs utilize live theatrical performances to promote healthy behaviors, leading to an increasing need to integrate evaluations of audience outcomes into such programs. In the present poster, we describe the evaluation of a benefit production of "The Vagina Monologues" presented by a university student chapter of V-Day, a national organization that uses arts-based events to promote awareness of violence against women and community engagement in anti-violence initiatives. Our poster presents findings from a participatory evaluation that applied narrative theory to examine audience member engagement. We discuss the design and implementation of an evaluation plan that balances financial constraints and students' needs while also assessing the value of current practices in achieving V-Day's goals. Successful strategies for gaining audience feedback will also be outlined with the hope that other arts-based organizations may be able to generalize these practices to evaluate audience impact.
Poster 194: A Multi-tiered Evaluation and Continuous Improvement Process for Technical Education Programs
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Christa Smith, Washburn University, christa.smith@washburn.edu
Debra Mikulka, Washburn University, debra.mikulka@washburn.edu
Gillian Gabelmann, Washburn Institute of Technology, gillian.gabelmann@washburntech.edu
Abstract: This poster presentation examines a multi-tiered evaluation and continuous improvement process for technical education programs utilized by a three-year, $20 million Department of Labor grant consisting of seven partnering institutions. The model assisted the grant program staff in identifying performance indicators and evaluation data to collect for procedural and programmatic improvement, as well as federal reporting requirements. As a model of continuous improvement, the process builds on the lessons learned and evaluation findings from the previous quarterly periods to identify shortcomings and aid with improving processes to better fill gaps. The strategies the grant management engaged in to assess the status of and improve program performance were evidence-based and involved program staff and consortium partners in collaborative revisions that increased understanding and consistency in reporting metrics. The poster presentation will also discuss the strengths, weaknesses, challenges, and potential for replicability of the continuous improvement process model.
Poster 20: Why Should Anyone Share? Is It Possible to Design Ethical and Effective Evaluation Systems to Capture Unpublished Evaluation Data
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Susan Snyder, Battelle Memorial Institute, snydersu@battelle.org
Lacy Fabian, Battelle Memorial Institute, fabianl@battelle.org
Chris Layfield, Battelle Memorial Institute, layfieldc@battelle.org
James Derzon, Battelle Memorial Institute, derzonj@battelle.org
Aaron Alford, Battelle Memorial Institute, alforda@battelle.org
Abstract: This presentation will orient participants to the problems unpublished data and proprietary evaluation data represent to evaluations that utilize systematic review or meta-analytic methodologies. Is it possible to responsibly collect and represent unpublished data? Why would an independent evaluator or practitioner release primary data to a third party? We will share a case-study of an unpublished data solicitation effort completed as part of a series of systematic reviews completed for the CDC Laboratory Medicine Best Practices project. We will discuss 1) social networking strategies for reaching data owners, 2) data-sharing mechanisms and 3) potential reward mechanisms.
Poster 14: Insightful Conversations: Using Solution-Focused Dialogue Skills in Evaluation Practice
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Emily Spence-Almaguer, University of North Texas, emily.spence_almaguer@unthsc.edu
Rita McWaters, The Project People, rita@theprojectpeoplellc.com
Abstract: Communication skills are vital to the practice of evaluation, particularly in regards to engaging stakeholders, understanding their informational needs and collecting qualitative data. This session will provide an introduction to the use of solution-focused dialogues in evaluation planning and data collection. Solution-focused dialogues are rooted in the human services, but have broad applicability outside of an intervention context. Solution-focused approaches are based on the assumption that people have the capacity to identify and successfully pursue the positive changes in their lives. Solution focused dialogues are unique in that they are almost entirely built around the construction of questions that stimulate reflection and planning. Gaiswinkler and Roessler (2009) refer to this as the expertise of not-knowing. This skill-building session will include a description of the principals of solution-focused dialogues, a comparison and contrast of actual interview transcripts using both solution-focused and traditional interview questions, and practice exercises using solution-focused dialogue tools.
Poster 147: Evaluation Knowledge and Skill Building in the 21st Century: The Canadian Evaluation Society (CES) Journey
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Terry Spencer, London District Catholic School Board, tspencer@office.ldcsb.on.ca
Brenda Stead, Stead and Associates & University of New Brunswick Canada, steadconsultants@gmail.com
Abstract: A key strategic priority of the Canadian Evaluation Strategy (CES) is to provide various knowledge mobilization platforms and networking opportunities (e.g., workshops, webinars, journal, conferences) through which evaluation practitioners and researchers learn about developments and innovations in evaluation theory, practice and research. Qualitative and quantitative data derived from various sources (e.g., member needs assessment surveys, informant interviews, stakeholder input) have served to inform the development of strategic and operational plans, related structures, processes and policies. This poster presentation will outline the CES's journey towards fulfilling this mandate of evaluation knowledge and skill building. We want to impart what has been done in the past few years to position CES as a global leader in evaluation capacity building in the 21st Century. We would like to share our journey we have embarked on - lessons learned, enablers, challenges, accomplishments and future directions to promote evaluation as a profession in Canada.
Poster 144: A Retrospective Analysis of Storyteller Children's Center: How to Reduce the Gap Between Knowledge and Practice Using a Knowledge Translation Toolkit
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Rubayi Srivastava, Goodland Assessment & Partnering Systems [GAPS], rubayi.srivastava@gmail.com
Michelle Enriquez, Goodland Assessment & Partnering Systems (GAPS), michelle.m.enriquez@gmail.com
Abstract: Storyteller Children's Center (STCC) provides early childhood education for homeless and at-risk children. STCC received five years of Kindergarten Report Cards of three small cohorts - through Desired Results Developmental Profile (DRDP), Peabody Picture Vocabulary Test (PPVT), Number Knowledge Test (NKT), and Kindergarten Student Entrance Profile (KSEP). Does evaluation end with four types of quantitative outputs? We developed a Knowledge Translation Toolkit to make meaning out of large amounts of data for those 5 years and expanded the evaluation to a retrospective analysis of 25 years of operations. We integrated several approaches including Most Significant Change (MSI), Appreciative Inquiry (AI), After Action Review (AAR), Impact Logs, and Horizontal Evaluation (HE) to create an interactive visual presentation of successes and failures and establish a framework for 1) increasing its capacity to manage and use knowledge, 2) better integrating evidence into decision-making, and 3) improving collaboration using an alignment model.
Poster 108: A Content Analysis of the Use of Mixed Methods in Public Health Research: A Mixed Methods Study
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Maria Stack, Virginia Tech, mstack@vt.edu
Elizabeth Creamer, Virginia Tech, creamere@vt.edu
Abstract: Over the past two decades, mixed methods research has become increasingly popular in the field of public health. Historically, public health researchers have held randomized controlled trials as the standard for evaluating health interventions; however, recently researchers have advocated for the addition of qualitative methods in order to more fully understand topics within the public health field. Because of the acceptance of qualitative methods in the field, mixed methods research has also gained popularity in public health. Thus, the purpose of the present study is to analyze methodological characteristics, such as design, mixing, and quality, of mixed methods research publications appearing in national and international journals in the field of public health. To investigate this, a sequential mixed methods content analysis, with a quantitative priority, will be used to analyze articles from three top tier public health journals.
Poster 36: Evaluating Organizational Support to Farmers Markets and Direct Marketing Farmers Using Survey Data and Social Network Analysis
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Michael Steketee, Westat, michaelsteketee@westat.com
Sujata Dixit-Joshi, Westat, sujatadixit-joshi@westat.com
Eric Williams, United States Department of Agriculture, eric.williams@fns.usda.gov
Abstract: Results are presented of Westat's national survey of farmers market managers and direct marketing farmers, a component of the Farmers Market Operations Study, using social network measures and visualizations. The United States Department of Agriculture (USDA) sponsored the Farmer Market Operations Study to better understand the environment of farmers markets and to increase access to farmers markets for participants in the Supplemental Nutrition Assistance Program (SNAP). One research question in this study was: what organizations serve farmers markets and direct marketing farmers, and what services do they provide? Survey respondents were asked what organizations have provided their farm or market with support and what organizations have provided support for incentives offered by the market or farm. We used egocentric social network analysis to measure and visualize reported support relationships. Findings suggest highly localized support networks for both farmers markets and direct marketing farmers. Methods, results, and limitations will be presented.
Poster 10: The Dilemma for Educational Evaluation: Where China and America Meet
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Hongling Sun, University of Illinois at Urbana-Champaign, hsun7@illinois.edu
Ruofei Tian, Shenyang Normal University, rftian@163.com
Jennifer Greene, University of Illinois at Urbana-Champaign, jcgreene@illinois.edu
Abstract: Globalization has generated increasing demand on education at all levels as global competitiveness is being considered as a critical determinant of national economic competitiveness (Ryan, 2010). To meet this demand, educational reforms are being undertaken in different countries -- east and west, developing and developed, centralized and decentralized. To support the accountability of national economic competitiveness, standards-based assessment is necessary; to cultivate individual well being, individually-focused multiple approaches of assessment and evaluation are crucial. How to balance and integrate these two forms of educational evaluation together? This is the dilemma for educational evaluation where China and America meets. The panel members will focus on the diverging changes of the assessment and evaluation policies in both nations in early 21st century, and then attempt to tackle the balancing issue, by involving the audience, to discuss who actually benefits from the educational evaluation and how.
Poster 155: Exploring AQUAD 7: A Free and Open-Access Computer Assisted Qualitative Data Analysis (CAQDAS) Tool in Qualitative Data Analysis
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Emma Sunnassee, University of North Carolina at Greensboro, emsunnas@uncg.edu
Lindsey Varner, University of North Carolina at Greensboro, l_dunn@uncg.edu
Holly Downs, University of North Carolina at Greensboro, hadowns@uncg.edu
Abstract: As technology evolves, we as qualitative data analysts are presented with multiple platforms to call upon to conduct our analyses, one of which is open access (i.e., free of charge) computer assisted/aided qualitative data analysis software (CAQDAS). The poster purposes to illustrate the features of AQUAD 7, a tool assists in the analysis of textual, audio, and video data. AQUAD 7 surpasses other tools of this genre, in that it can perform quantitative analyses and directly code media by allowing the user to code audio and video on a timeline (Huber & G++rtler, 2013). Furthermore, AQUAD 7 has extensive qualitative data analysis capabilities that enable the organization, categorization, analysis, and synthesis of data (Huber & G++rtler). This poster provides an overview of AQUAD 7 by demonstrating its capabilities, limitations and benefits. In turn, this will provide graduate students and experienced evaluators alike an opportunity to see the advantages and challenges of employing AQUAD 7.
Poster 30: Evaluating Social Services Trainings: Opportunities to Make Multi-Level Improvements
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Suzanne Sutphin, University of South Carolina, sutphist@mailbox.sc.edu
Melissa Strompolis, University of South Carolina, strompol@mailbox.sc.edu
Abstract: The Center for Child and Family Studies (CCFS) at the University of South Carolina has a long-standing relationship with the Department of Social Services (DSS). Part of that relationship involves providing social service-related trainings to DSS social workers. This poster will display information on evaluations of social service-related trainings conducted by CCFS, specifically child welfare basic and adult protection services trainings. The evaluation information provides a unique look at the impact of variations of training length (e.g., three weeks versus six weeks), frequency of assessment (e.g., one, three, and four time points), assessment type (e.g., pre-test versus retrospective pre-test), and source of assessment (e.g., social worker and supervisor). The findings from these evaluations of social services trainings are useful to examine the most effective training formats, specific areas for which to make training modifications, and designs of evaluations that produce the most useful data.
Poster 77: Measuring the Impact of the YMCA's Centres of Community: Incorporating Complexity and Context
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Carrie Tanasichuk, YMCA of Greater Toronto, carrie.tanasichuk@ymcagta.org
Sarah Earl, YMCA of Greater Toronto, sarah.earl@ymcagta.org
Cindy Duval, YMCA of Greater Toronto, cindy.duval@ymcagta.org
Michael Hall, YMCA of Greater Toronto, michael.hall@ymcagta.org
Virginia Dimoglou, YMCA of Greater Toronto, virginia.dimoglou@ymcagta.org
Abstract: The Vision of the YMCA of Greater Toronto is that our communities will be home to the healthiest children, teens and young adults. In order to meet this goal, we have adopted a "Centres of Community" model. Traditionally, the YMCA has operated many of its programs in separate locations. The Centre of Community model involves co-locating our programs together wherever possible according to the particular needs of a community. The impact of our Centres involves the combined effect of our individual programs, the partnerships we develop in our communities and our thought leadership and advocacy. Not only do we want to assess the impact each Centre has on YMCA members, but we will also be assessing what impact the facility has on the health of the surrounding community. As such, this work will contribute to determining how to evaluate complex, place-based programming using multiple strategies.
Poster 8: Addressing Student Needs: School District Approaches to Evaluating Programs for Special Student Populations
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Erika Taylor, National Education Association, taylorerika1@gmail.com
Carole Keane, Prince George's County Public Schools, carole.keane@pgcps.org
Cecily Adams, Prince George's County Public Schools, cecily.adams@pgcps.org
Margaret Patterson, Research Allies for Lifelong Learning, margaret@researchallies.org
Abstract: To date, the 21st century has been characterized by major societal shifts, including increased ethnic diversity and economic downturn. In many public school districts, these larger shifts have been reflected in the unique and changing needs of their students, and increases in programs to support these populations. This poster focuses on three programs that serve a growing population of vulnerable students–homeless students, those in informal kinship care, and students who are identified as being on the autism spectrum–in a large school district in Maryland. The poster will highlight how the findings from the evaluation of each program were used to improve, expand or better target services to these populations.
Poster 199: Moderate Alcohol Consumption and Health: An Evaluation of the impact on Alcohol Prevention Practice
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Gisele Tchamba, Western Michigan University, gisele.tchamba@wmich.edu
Abstract: There is a general agreement that excessive drinking is responsible for enormous disruption in social, psychological, and physical well-being. Alcohol consumption is regarded as the third most preventable cause of death and alcohol prevention policy has been focused on reducing excessive drinking. Recently, significant segment of epidemiological literature reveals that moderate alcohol consumption has important health benefits including reduced mortality rate from coronary heart diseases (CHD) and others. Conversely, other epidemiological literature finds significant health risks with alcohol consumption regardless of the quantity. The purpose of this evaluation is to explore the health risks of moderate drinking for prevention policies. It uses qualitative approach to identify and assess prevention specialists at an inner city program and how they respond to the current issue. Precisely, how does this discourse influence their work and what is their understanding of moderate drinking? What guidelines are available to help them educate the population?
Poster 122: Social Network Analysis in Program Evaluation: Evaluating Interactions Over Time in a Nation-wide Network of Students and Faculty in Plant Breeding
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Mao Thao, University of Minnesota, thao0181@umn.edu
Frances Lawrenz, University of Minnesota, lawrenz@umn.edu
Abstract: The use of social network analysis is growing in the field of program evaluation - particularly as evaluations of systems are becoming more common. This poster presentation highlights a program evaluation that uses social network analysis methods to assess interactions among its participants over three years of programming. It outlines and illustrates how evaluators can use social network analysis methods and what kinds of information social network analysis can provide. Two-mode network data were collected through annual surveys of student and faculty participants of the Triticeae Coordinated Agricultural Project (TCAP), a nation-wide effort to improve the quality of wheat and barley breeding and increase the number of plant breeders through research and education. Social network analysis was completed using the UCINET software to examine interactions within the network over time. Findings of network cohesion over time are discussed, as well as implications for the use of social network analysis in evaluation.
Poster 218: How Can Evaluation Inform Policy on Tailored Programming?
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Amanda Thompkins, King County, amanda.thompkins@kingcounty.gov
Janet Salm, King County, janet.salm@kingcounty.gov
Abstract: Policymakers today are more data-driven. They increasingly turn to internal government evaluators to assess the value not simply of programs, but of investment portfolios and the strategies contained therein. Many policy areas include strategic investments in programs tailored to meet the needs of specific sub-populations such as as cultural groups, veterans, or young adults. However, members of these groups are also clients of general programs. How can evaluators help policy-makers decide whether the data supports making additional investments in tailored programs? Do participants in tailored programs differ in their trajectories from those in general programs, and if so, what does this tell us about the value of tailored programs? This session will discuss a range of evaluative approaches to understanding tailored programs, using results from an evaluation of youth and young adult homeless housing programs as an exemplar.
Poster 71: Inter Actions Among Program Evaluations, High-Leverage Activities, and Organizational Learning
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Bill Thornton, University of Nevada, Reno, thorbill@unr.edu
George (Gus) Hill, University of Nevada, Reno, gushill@gmail.com
Janet Usinger, University of Nevada, Reno, signerj@unr.edu
Jacque Ewing-Taylor, University of Nevada, Reno, jacque@unr.edu
Abstract: Abstract The pressure to improve the operations of our social organizations has increased geometrically during the past few years. Knowledge, organizational learning, and program evaluations are interrelated with effective leadership of complex social organizations. In these organizations, high-leverage activities are critical to promote organizational learning and continuous improvement; however, effective leadership is often limited by functions, policies, and structures. This paper will briefly summarize high-leverage leadership actives and discuss how they affect organizational improvement. Specifically, this paper will analyze how systemic approaches, systems thinking, and program evaluations can identify high-leverage activities and promote organization change. The interactions among evaluations, existing knowledge, developed knowledge, effective communication, and leadership will be illustrated. Methods by which organizations might develop structures and procedures to promote application of high-leverage activities, through effective evaluations will be presented and specific examples will be provided.
Poster 165: Evaluation Practice: An Overview of Evaluation at the National Institute of Allergy and Infectious Diseases
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Krystal Tomlin, National Institutes of Health, tomlinka@mail.nih.gov
Liberty Walton, National Institutes of Health, liberty.walton@nih.gov
Dione Washington, National Institutes of Health, dione.washington@nih.gov
Guy Arcuri, National Institutes of Health, garcuri@mail.nih.gov
Brandie Taylor, National Institutes of Health, taylorbr@mail.nih.gov
Abstract: The Strategic Planning and Evaluation Branch (SPEB) within the Office of Strategic Planning, Initiative Development, & Analysis (OSPIDA) of the National Institute of Allergy and Infectious Diseases (NIAID), part of the National Institutes of Health (NIH), conducts a wide breadth of evaluations, focusing on the assessment of both scientific and administrative programs. Recognizing that there is not one evaluation method that is well-suited to the wide-range of evaluations within SPEB's purview, SPEB staff assess each evaluation question to determine the best approach. This poster provides an overview of the types of evaluations conducted within NIAID, as well as an illustration of the methods used and the challenges and value of each. The information presented demonstrates how evaluation is conducted in the 21st Century at NIAID, and provides sample methods for approaching a variety of evaluation types.
Poster 73: Measuring Individual and Organizational Evaluation Capacity Building - California's Tobacco Control Programs
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Jeanette Treiber, University of California at Davis, jtreiber@ucdavis.edu
Robin Kipke, University of California at Davis, rakipke@ucdavis.edu
Catherine Dizon, University of California at Davis, czdizon@ucdavis.edu
Diana Cassady, University of California at Davis, dlcassady@ucdavis.edu
Abstract: Taylor-Powell and Boyd (2008) suggest a simple logic model for Evaluation Capacity Building (ECB) that links efforts to individual, team, program, and organizational change. The Center for Evaluation and Research at UC Davis measured two of these, individual and organizational change, of approximately 120 individuals representing 70 organizations that perform tobacco control programs. Between March and September 2013, CER provided training through three training events: 1. local onsite training in evaluation report writing, 2. a webinar on conducting intercept surveys, and 3. a webinar on conducting key informant interviews. We measured individual change through retrospective pre-tests for each of these three training events (Nimon et. al. 2011). Organizational change was assessed by comparing final evaluation report scores of the organizations with the scores from previous funding cycles (Treiber et al., 2013). The presentation's focus will be on the usefulness of retrospective pretests and report results trend analysis as ECB measures.
Poster 91: Improving Medical Education Curriculum: Fostering Evaluative Thinking
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Christine Trupin, Meharry Medical College, ctrupin@mmc.edu
Cassandra Ward, Meharry Medical College, cward@mmc.edu
Digna Forbes, Meharry Medical College, dforbes@mmc.edu
Etheleen Hill, Meharry Medical College, ehill@mmc.edu
Suzan Tropez-sims, Meharry Medical College, stsims@mmc.edu
Chang Chen, Meharry Medical College, ckchen@mmc.edu
Fatima Barnes, Meharry Medical College, fbarnes@mmc.edu
Carol Freund, Meharry Medical College, cfreund@mmc.edu
Abstract: A Curriculum Evaluation Task Force Committee in the School of Medicine was formed to assess the effectiveness of processes intended to ensure that the curriculum was adequate for medical training. The committee seeks to: assess the extent to which existing evaluation processes provide useful information of the curriculum adequacy in providing assessing coverage of key competencies in medical education; provides actionable data on the strengths and challenges that needed to be addressed to improve the quality of the education experiences; and facilitate the use the findings to refine the curriculum. Equally important is to develop an on-going evaluation process to guide program improvement and education enrichment. Using evaluation to improve educational experiences of trainees continues to be a challenge in educational programs. Multiple factor contributes to limited use. This study is based on assumption that facilitating evaluative thinking among practitioners can increased evaluation use to improve educational programs.
Poster 246: Evaluation of Local Governmental Change aia Descriptive Single-case Study
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Jennifer Uhrich, University of Southern California, uhrichjenny@gmail.com
Abstract: The purpose of this presentation is to graphically demonstrate the use of Descriptive Case Study methodology to evaluate local governmental change. In this case study the Santa Barbara Board of Supervisors hired Chandra Wallar as CEO in 2010 to limit budgetary shortfalls while preserving as many services as possible. She employed the strategy of consolidation of multiple Santa Barbara County Departments into a new Community Services Department. This example provides a critical case study evaluation: a model example of an emerging strategy for rectifying local governmental budget crises. Although Santa Barbara County continues to struggle with budgetary issues, the consolidation of departments dramatically decreased budgetary shortfalls while preserving services. This poster will present a descriptive single-case study model as a tool for evaluating local government consolidation and to discuss the methods pros and cons.
Poster 106: Measuring Health System Capacity for Transition of HIV Clinical Programs to Local Ownership: A Review of CDC/PEPFAR Efforts in Mozambique
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
N Ilona Varallyay, ICF International, ilona.varallyay@icfi.com
Janna Brooks, Centers for Disease Control and Prevention, zir7@cdc.gov
Charity Alfredo, Centers for Disease Control and Prevention, alfredoc@mz.cdc.gov
Mindy Hochgesang, Centers for Disease Control and Prevention, hochgesangm@mz.cdc.gov
Kebba Jobarteh, Centers for Disease Control and Prevention, jobartehk@mz.cdc.gov
Eric Sarriot, ICF International, eric.sarriot@icfi.com
Abstract: In the context of the ongoing transition of the effective management and oversight of CDC/PEPFAR HIV clinical programs to local ownership by local partners including ministries of health, the establishment of systematic approaches to monitor health system capacity and the transition process itself is needed. This paper presents the multi-dimensional mixed methods monitoring approach that was piloted in Mozambique in 2012. The focus of this exercise was to assess a) health system capacity outcomes at sub-national level across 10 health system domains and b) progress on the shifting roles of Implementing Partners and Provincial Health Directorates during the transition. To facilitate decision-making, these findings are presented alongside trend data on PEPFAR health outcome indicators. The paper also presents lessons learned on the design and implementation of assessment methods and provides recommendations to help ensure that methods used to collect and present findings generated are relevant, easily interpretable, and actionable.
Poster 151: Evaluation of a University's International Student Orientation Program
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Divya Varier, Virginia Commonwealth University, varierd@vcu.edu
Lisa Abrams, Virginia Commonwealth University, lmabrams@vcu.edu
Amanda Velez, Virginia Commonwealth University, advelez@vcu.edu
Abstract: This evaluation of an international student orientation was implemented in a large southeastern university serving about 1500 international students representing over 100 countries. The purpose of the orientation is to provide critical legal/immigration information and formally welcome new international students to the university and the United States. Guided by questions based on nine program goals, the evaluation involved a descriptive design with a mixed-method approach to data collection. Seven key staff members were interviewed and 154 undergraduate and graduate international students responded to an online survey about various aspects of their orientation experience. Interviews were transcribed and analyzed using the evaluation questions as a framework, and descriptive analyses of the survey results provided information about student perspectives. Findings are presented in the context of program goals and staff and student perceptions of the orientation's effectiveness. Evaluation considerations for programs and services for students in higher education are discussed.
Poster 112: Improving Through Evaluation Civil Society's Participation in the "XXI Century Socialism"
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Clemencia Vela, Evaluadores Ecuador Network, clemvela@gmail.com
Abstract: The so-called "XXI Century Socialism" has been established in some in Latin America countries: Ecuador, Venezuela and Nicaragua; which, according to some scholars, follows more closely the Cuban model rather than more modern social approaches. In this context, the paper offers recommendations of how evaluation can support common citizen's participation in policies improvement. It is based on lessons learnt from an evaluation of women and civil society's current participation. To explore the subject the paper will include results of: i) a survey done among common citizens, considering gender; ii) existing channels for participation; iii) evaluation of cases where common citizens have pursued policy and legal changes. The presentation would propose suggestions to improve public participation through evaluation, such as improve awareness to civil society and civil servants, involving public servants on evaluation networks, creation of working groups for evaluation, etc..
Poster 17: Agility, Flexibility, and Immediate Utility in Government Evaluation Practices: The AmeriCorps VISTA (Volunteers in Service to America) Training Assessment
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Nicole Vicinanza, JBS International Inc, nvicinanza@jbsinternational.com
Patrick Triano, Corporation for National and Community Service, ptriano@cns.gov
Peter Lovegrove, JBS International Inc, plovegrove@jbsinternational.com
Abstract: The use of evaluation findings is a central issue addressed by evaluators, particularly those engaged in research on federal programs designed to assist those in need. In such cases, the timing and ability to communicate about findings throughout the course of the evaluation can make the difference in evaluation use. The ability to share and use evaluation findings throughout the evaluation process is especially relevant for research on large scale government programs like the AmeriCorps VISTA Program, where considering findings throughout the process may help the program save time and money. This session illustrates the lessons learned when evaluators and clients engage in ongoing communications and with an eye towards creating an evaluation structure that is agile enough to anticipate and successfully incorporate unexpected findings, and yet maintains its rigor.
Poster 79: Are We Able to Better Estimate Impacts Through System Concepts?
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Pablo Vidueira, Technical University of Madrid, pablovidueira@gmail.com
José María Díaz-Puente, Technical University of Madrid, jm.diazpuente@upm.es
Abstract: The estimation of impacts is one of the most demanding activities in the program evaluations, but that is also one of the evaluation activities providing more useful information facing the improvement of program's implementation and design. Many difficulties arise because of the lack of methodologies able to address the complexity of the impacts generated by programs. In order to overcome that problem, new methodologies are needed, and "system thinking" and one of its methodologies: system dynamics seem to have great insights on how to improve program evaluation. The aim of this paper is to combine the huge efforts made in the theoretical development of these tools and also in its implementation in case studies to apply all in a real evaluation with time, data, budget and legal constrains. It would let us know to what extent these methodologies help the evaluator and the program designer to do their work better.
Poster 100: Preparing for Health Care Reform: The Impact of Orange County's Low Income Health Plan on Health and Service Indicators in Uninsured Adults
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Shelley Vrungos, Orange County Health Care Agency, svrungos@ochca.com
Sepideh Saghari, Orange County Health Care Agency, ssaghari@ochca.com
Timothy Allison-Aipa, Orange County Health Care Agency, tallison@ochca.com
Abstract: The Medical Services Initiative Program (MSI) has served as Orange County's safety net program providing health care to low income uninsured adults since 1983. Under a federal 1115 waiver (2007), coverage was expanded from episodic (urgent/emergent) to primary and preventive care improving access to high quality, coordinated care. In the most recent 1115 Medicaid Waiver (2010), the Low Income Health Plan (LIHP), which has included several patient-centered program innovations, has had a goal to increase uninsured adults' access to comprehensive health care services in preparation for health care reform implementation in 2014. An evaluation of the LIHP on access, health, utilization and patient experience outcomes was conducted. Results indicate that the LIHP has reduced rates of hospitalization and emergency department visits while increasing visits to assigned primary care "medical home" providers. Self-reported patient satisfaction with medical home providers, access to care, and the LIHP program overall is high.
Poster 103: Addressing Multicultural Evaluation Challenges: University Programs Offered for International Students
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Dammika Lakmal Walpitage, The University of Tennessee, dwalpita@utk.edu
Niranji Pathirage, The University of Tennessee, niranjipathirage@gmail.com
Abstract: Establishing a formal evaluation mechanism is an essential task for programs that are targeted for international students. However, designing evaluations for such programs with multinational participants, and multinational program staff who have less experience with evaluation is a challenging and stressful task. This presentation will discuss our experiences in conducting evaluations with service providers for international students at a large southeastern university. We will highlight the importance of establishing an evaluation mechanism for such programs, as well as the difficulties we encountered and lessons learned when conducting a series of evaluations. Moreover, we will provide suggestions and strategies for novice evaluators on how to address multicultural evaluation challenges when developing a formal evaluation mechanism for programs with no prior evaluation experience. Additionally, strategies for increasing stakeholder engagement and building client evaluation capacity will be discussed.
Poster 27: Concurrent Mixed Methods With Diverse Stakeholders: Pilot Course Evaluations in Online Higher Education
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Alice Walters, Walden University, alice.walters@waldenu.edu
Christy Fraenza, Walden University, christy.fraenza@waldenu.edu
Shanna Van Ness, Walden University, shanna.vanness@waldenu.edu
Abstract: Academic settings present evaluation challenges to address the perspectives of multiple stakeholders. Concurrent mixed methods evaluation designs are viable approaches in the educational context presenting several advantages. Cost effectiveness, efficiency, and timely results are outcomes of concurrent quantitative and qualitative methods. This poster will discuss using concurrent mixed methods to evaluate two different pilot courses in online higher education. Evaluation of an online graduate writing course with an embedded teaching assistant/peer mentor and an online doctoral literature review course piloting an online whiteboard with a teaching assistant/peer mentor are assessed. The evaluation design is an inclusive strategy to integrate perspectives of diverse course participants of faculty, students, teaching assistants, and administrators.
Poster 168: Diversity Training Grants Analysis at the National Institute of Allergy and Infectious Diseases
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Liberty Walton, National Institutes of Health, liberty.walton@nih.gov
Krystal Tomlin, National Institutes of Health, tomlinka@mail.nih.gov
Dione Washington, National Institutes of Health, dione.washington@nih.gov
Guy Arcuri, National Institutes of Health, garcuri@mail.nih.gov
Brandie Taylor, National Institutes of Health, taylorbr@mail.nih.gov
Abstract: The National Institutes of Health (NIH) uses several grant mechanisms that aim to increase the diversity of the biomedical scientific research workforce. The National Institute of Allergy and Infectious Diseases (NIAID), part of NIH, promotes training of a diverse workforce through Diversity Supplements and F31 Diversity Awards, which target scientists from underrepresented racial and minority groups, and those with disabilities or from disadvantaged backgrounds. Through the provision of training grants, NIAID anticipates that awardees will continue to apply for and receive future funding from NIH, thereby increasing the continued participation of a heterogeneous population in the NIH scientific research community. To determine if these grant mechanisms are achieving this goal, NIAID's Strategic Planning & Evaluation Branch/OSPIDA and Division of Extramural Affairs conducted an outcome evaluation of the Diversity Training Awards. This poster illustrates the evaluation methodology, and provides key study findings, as well as lessons learned from the evaluation process.
Poster 86: The Impact of Alice Workshops on Teachers' Professional Growth: The Mixed-Methods Evaluation of An Innovative Technology Experiences for Students and Teachers (ITEST) Project
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Yuanyuan Wang, University of Pittsburgh, yuw21@pitt.edu
Bryan Stephany, University of Pittsburgh, bms65@pitt.edu
Abstract: Duke Scale-up Innovative Technology Experiences for Students and Teachers (ITEST) provided K-12 teachers in North Carolina and South Carolina with summer workshops run by local college faculty to promote incorporation of Alice instruction into teachers' STEM content classes. The Collaborative for Evaluation and Assessment Capacity (CEAC) adopted a mixed methods approach to measure the impact of summer workshops on teachers' content knowledge and skills in Alice, teachers' feedback of instructions and their fellow participants, college faculty's feedback of teachers' lessons, and teacher utilization of Alice in their classroom. Findings indicated that teachers substantially increased content knowledge and skills in Alice and integrated Alice software in their classroom. Teachers were satisfied with the instruction, and college faculty and fellow teachers offered positive comments on the lessons of teachers. Our evaluation findings provide implications for other programs with similar missions to increase capacity to use programming concepts within traditional K-12 STEM courses.
Poster 104: Toward a Culturally Competent Method of Evaluating Rural Teachers
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Michaele Webb, Syracuse University, miwebb@syr.edu
Abstract: Although all rural areas are unique, rural primary and secondary school teachers face a number of similar obstacles, such as high poverty rates, in the context that they teach in (Fowler, 2012). The AEA Public Statement on Cultural Competence in Evaluation states that evaluators must be aware of the history and values of the group that is being evaluated and collect information regarding the people and context of the evaluation (American Evaluation Association, 2011). This poster will propose factors that must be considered when evaluating the performance of teachers in rural school districts. American Evaluation Association. (2011). Public Statement on Cultural Competence in Evaluation. Fairhaven, MA: Author. Retrieved from www .eval .org. Fowler, R. H. (2012). Rural characteristics and values: A primer for rural teachers from non-rural backgrounds. National Teacher Education Journal, 5(4), 75-80.
Poster 32: A Logic Model for the Development of Performance Indicators to Evaluate Diverse Clinical Research Programs in Domestic and Global Settings
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Susanna Weiss, National Institutes of Health, weisssu@niaid.nih.gov
Laura McNay, National Institutes of Health, lmcnay@niaid.nih.gov
Abstract: This logic model describes how one division at the National Institutes of Health is developing performance measures to provide data-driven progress reports for programs and employees engaged in the facilitation and administration of clinical research in domestic and global settings. This road map describes the rationale and multi-factorial, incremental research forming the development, validation and implementation of Key Performance Indicators that reflect the work-product of the division's branches and special projects. Fundamental components are: (1) contextual information about factors and intervening variables that may impact performance; (2) identification of comparison measures, e.g., baseline performance data and leading industry/government benchmarks against which performance can be compared; (3) proposed data collection sources, methods, instruments and analysis, and (4) tools for assembling and tracking performance metrics. The model also emphasizes the importance of participatory methods of building performance metrics and goal-setting benchmarks to ensure that they adequately and fairly reflect each worker's productivity.
Poster 107: CTSA Evaluation is No Piece of Cake: Evaluators' Recipes for Learning About Clinical and Translational Science Award Institutes
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Katelyn Westaby, University of Wisconsin at Madison, kwestaby@wisc.edu
Valerie Moody, The University of Iowa, valerie-moody@uiowa.edu
Abstract: Evaluators entering the world of Clinical and Translational Science Award (CTSA) institutes are often struck by the complexity of evaluating these institutions. Each CTSA encompasses several cores, or research service providers (i.e., drug discovery and development, biomedical informatics, or community engagement), with drastically different focuses and, therefore, indicators of progress. Annually, evaluators are challenged with developing integrated reports for the National Institute of Health using data collected in the diverse CTSA cores. This poster presentation will first display how two new-to-CTSA evaluators tackled the task of understanding CTSAs. The presenters will share their introspective feedback regarding learning effectiveness for tasks such as networking, conference calls, previous grant application reviews, and site visits. To supplement, the authors will present survey results from other new-to-CTSA evaluators regarding their promising practices for understanding evaluation within these complex systems.
Poster 163: The Michigan Asthma Adjustment: Adjusting Evaluation Findings to Account for Natural Improvement of Asthma Symptoms Over Time
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Maggie Wilkin, University of Michigan, mwilkin@umich.edu
Laurie Carpenter, University of Michigan, lauriemc@umich.edu
Peter XK Song, University of Michigan, pxsong@umich.edu
Yi-An Ko, University of Michigan, yianko@umich.edu
Noreen Clark, University of Michigan, nmclark@umich.edu
Abstract: Symptoms and health care use of children with asthma often improve with age even in the absence of a targeted intervention program; without comparison groups, evaluations of childhood asthma interventions overestimate the effect on outcomes. Using data from three controlled trials, mixed-effects models were utilized to describe the developmental pattern of symptoms and health care use in children with asthma; symptoms in children under the age of 10 declined about 20% each year and emergency department visits declined with age regardless of baseline symptom frequency. The Michigan Asthma Adjustment was developed to account for this natural improvement in studies with only pre-post data. The adjustment uses a bootstrapping technique to create pseudo-controls matched to intervention subjects on demographic characteristics, such as race, age and gender. The creation of the adjustment and its application to a sample of children in a multi-site, statewide childhood asthma intervention program will be presented.
Poster 28: A Multi-Site Collaborative Model for Assessing Professional Competencies
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Beverly Williams, University of Alberta, beverly.williams@ualberta.ca
Stacy Grainger-Schatz, University of Alberta, stacy.grainger-schatz@ualberta.ca
Abstract: The University of Alberta Faculty of Nursing Baccalaureate program is a unique nursing education program. It is a collaboration among four geographically distance sites all of whom implement a common curriculum. Comprehensive evaluation of the program takes place over 4 -+ years, beginning with program entry and ending six months after graduation. Professional competencies are used as the basis for program evaluation. The Collaborative program was recently awarded the maximum accreditation by the National Accreditation Association for the discipline. The program evaluation strategy has been a key component since the beginning of the Collaborative program. The data that we have collected from students, faculty, preceptors, graduates and employers over time has helped shape the current curriculum. The undergraduate program evaluation model is currently being adapted for the graduate program.
Poster 110: Linking Science Programs Inputs and Economic Outcomes: A Hybrid Model of the Payback and Public Value Mapping Frameworks
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Jeffrey Williams, CRDF Global, jwilliams@crdfglobal.org
Abstract: Evaluators and other analysts have long struggled to establish direct links between inputs for science programs and economic outputs and outcomes. In today's political and fiscal climate, there are increasing calls for evidence of those links in order to justify science program funding. Unfortunately, for well-known methodological reasons, the proper data needed to answer such questions remains elusive. This poster will provide evaluators with an accessible and methodologically robust framework for linking science program inputs with reportable economic outputs via a hybrid of the Payback and Public Value Mapping frameworks. There will be a specific focus on guidelines for extracting reliable economic outcome proxies for situations in which economic outcomes were not included in the original goals and objectives of a science program.
Poster 216: We Need a Class: a Sexual Education/Health Needs Assessment of Undergraduate Women of Color
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Tamara Williams Van Horn, University of Colorado at Boulder, tamara.williams@colorado.edu
Tamara Williams Van Horn, University of Colorado at Boulder, tamara.williams@colorado.edu
Tamara Williams Van Horn, University of Colorado at Boulder, tamara.williams@colorado.edu
Abstract: This study analyzed focus group data during a needs assessment of underrepresented undergraduate women attending a predominantly white institution, around sexual health and sexual education experiences. Young women of color continue to be a high-risk group for unwanted pregnancies and sexually transmitted infections (STIs), and the campus Public Health Service aims to develop culturally-relevant messaging (CDC 2009). While cognizant of the threat STIs pose to young adults in general, and college students specifically, these women consistently employed a distancing strategy of narrating the needs of various communities they identify with, while erasing themselves from these narratives. Results of this study will inform changes to existing university-delivered reproductive and sexual health content to align with the needs and capacities of underrepresented students. They also lay a framework for a culturally responsive content evaluation checklist applicable to college populations. This poster reports focus group findings, and suggests areas for future research.
Poster 118: Meet Them Where They Are: Web-based Evaluation Capacity Building
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Lori Wingate, Western Michigan University, lori.wingate@wmich.edu
Krystin Martens, Western MIchigan University, krystin.s.martens@wmich.edu
Abstract: This poster will highlight the work of EvaluATE, a National Science Foundation-funded evaluation resource center. Primarily through webinars, EvaluATE works to build evaluation capacity among a diverse audience that includes NSF program grantees, internal and external evaluators, researchers, grant writers, and college administrators. With such a range of participants, EvaluATE uses this venue to educate and engage both evaluators and evaluation consumers, which has the added of benefit of fostering the use of common evaluative language and concepts-a perennial challenge in most evaluation contexts. EvaluATE webinars and other resources are free and publicly accessible, representing an additional option for professional development for the global evaluation community. We will share what we've learned using webinars as a platform for teaching, including the results from our internal and external evaluations, and invite recommendations about concepts and competencies that are essential to address in this context.
Poster 172: Evaluating an Alternative Food Project in Low-Resource Chicago Neighborhoods
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Katherine Wright, Northwestern University, k-wright@northwestern.edu
Lauren Anderson, Northwestern University, lauren-taylor@northwestern.edu
Abstract: Launched in 2012, the Neighbor Carts Program promotes the opportunity for economic success and healthy food access through an unconventional retail structure. Neighbor Carts are independent produce carts that operate in underserved areas throughout Chicago. Detailed analysis of pilot year data focuses on the relationship between each cart and its surrounding community by examining consumer behavior patterns, workforce development efforts, and the economic sustainability of the model. Preliminary results suggest that different persons interact with food in different ways, reflecting complex, spatially dynamic foodscapes. Carts in close proximity, but operating in different neighborhoods, expressed wholly unique patterns. Results also suggest that food carts near community centers and/or transportation hubs were the most successful in distributing produce to low food-access areas. Evaluating hyper-localized patterns of food access at alternative food projects may be effective in better assessing a complicated, local foodscape.
Poster 42: Community Arts Outcomes
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Rebecca Yenawine, Maryland Institute College or Art and New Lens, rebeccayenawine@newlens.info
Abstract: In community art, the art-making process and the final product are designed to affirm, strengthen, and transform individuals and/or community. Though the idea of art produced in community to celebrate individual and community strengths occurred in many ancient cultures, community art as a field is young. It has only been in the last ten years that degree programs have emerged. As a result, there is little empirical research that describes its impact. The research in this presentation includes a description of two qualitative studies with practitioners and participants, which lay the foundation for understanding arts processes in a community context. Preliminary assessment models and how art can be used strategically as a tool for youth development and community building will be explored.
Poster 25: Doing More With Less: Streamlining Data Collection, Data and Project Management, and Resource Scheduling in a iOS-Compatible Application
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Susan York, University of Hawaii at Manoa, yorks@hawaii.edu
Susan Saka, University of Hawaii at Manoa, ssaka@hawaii.edu
Abstract: Conducting evaluations in the current economy means doing more with less. It is no secret that evaluators need to work more efficiently without jeopardizing accuracy or quality, and that suggests making the most of current technology. Our assessment targets are 20,000 secondary students and 575 preschoolers in 144 schools on six islands in a three-month assessment window for three different projects. The evaluation team consists of nine professional staff and six college student assistants with a total of 7.0 FTE, working on 1-3 projects. Juggling evaluator and school schedules and tracking the various assessments, incentives, permission forms, and travel arrangement requires flexible database software that synchronizes with iOS devices. Find out what did and did not work and how we adapted. Our lessons learned can be your list of do's and don'ts as you make the move to integrating technology into evaluation.
Poster 49: An Assessment of Goal-Free Evaluation: Case Studies of Four Goal-Free Evaluations
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Brandon Youker, Grand Valley State University, youkerb@gvsu.edu
Lyza Ingraham, Grand Valley State University, ingrahaa@mail.gvsu.edu
Abstract: This article provides a critical review of four goal-free program evaluations. Goal-free evaluation (GFE) is a lesser used evaluation model whereby the independent evaluator is intentionally screened from the program's stated goals and objectives in hopes of reducing perceptual bias. The findings from these case studies are focused in three areas: (1) elements of the programs evaluated and the evaluation contexts (e.g., types of programs, pre-evaluation conditions, size of evaluation budgets), (2) the design of the GFEs (e.g., screening methods, data collection methods), and (3) the expertise of the goal-free evaluators (e.g., GFE-specific training, graduate degrees attained). With these evaluations, GFE served as a qualitative data collection method or tool, and all of the GFEs were relatively small in size, scope, and budget. Conclusions call for further operationalization of GFE if its use is to increase as well as for systematic and empirical study comparing GFE with goal-based evaluation.
Poster 204: The Use Gendered Innovations Checklists in Universities
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Justin Young, University of Houston, jdyoung4@uh.edu
Abstract: Gender has historically been overlooked in scientific research and Schiebinger et al. (2011-2013) developed checklists to address this. For example, women were excluded from drug trials in the 1970s and 1980s which arguably led to drugs that were harmful to women being released in the 1990s (United States Government Accountability Office, 2001). This is one reason why Schiebinger et al. developed checklists for engineering, health and medicine, as well as tissues and cells were developed. This research looks at how these checklists are being used in university settings. Professors' and students' views of gendered innovations checklists are also presented.
Poster 97: Multi-level Evaluation Tool for Estimating Business Interruption Due to Seismic Damage
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Meagan Young, University of Minnesota, youn1479@umn.edu
Judith Mitrani-Reiser, Johns Hopkins University, jmitrani@jhu.edu
Abstract: Various surveys have been developed by sociologists to better understand the immediate and long-term effects natural disasters have on society. These surveys investigate business interruption, which causes financial stress on businesses and hardship on a community in the affected region. This study focuses on developing surveys for use by earthquake reconnaissance teams in assessing the initial physical damage and economic impact caused by an earthquake. The surveys will capture metrics including global/regional damage impacts, individual building damage, and business-specific losses. Follow-up surveys may be conducted with businesses to determine disaster recovery and economic issues, length and extent of closures, and other economic fallout. This data can be compiled into simulation programs by engineers to illuminate prominent variables for predicting and minimizing the impact and recovery of structural, nonstructural, and economic systems.
Poster 249: Using a Collaborative Evaluation Approach to Assess the Impact of Recreational Programs on the Well Being of Older Adults
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Tiffany L Young, University of South Florida, tlyoung@mail.usf.edu
Anh Kellermann, University of South Florida, napham@mail.usf.edu
Brittany  Stowers, University of South Florida, stowersb@usf.edu
Abstract: Older adults are one of the most diverse and fastest growing populations in the United States. These individuals are concerned about their well being; therefore, they are active and demand engaging community programs. Wellness programs that include physical activity and social engagement have been documented to improve the overall well being of older adults. Organizations that serve older adults have implemented innovative wellness programs designed to promote health, education, recreation, socialization, and creative expression for these individuals; and they want to know the effectiveness of these programs. The focus of this poster is on a collaborative evaluation that used a mixed methods design to assess the affects of a recreational centers programs on the perceived physical, mental, and social health outcomes of its older adult patrons. This poster will illustrate the methods and findings associated with the evaluation.
Poster 252: Whats in it for Me? Using Cross-site Evaluation Data for Local Evaluation
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Sarah Yuan, University of Hawaii at Manoa, cwlau@hawaii.edu
Turro Wongkaren, University of Hawaii at Manoa, turro@hawaii.edu
Abstract: Cross-site evaluation has become a common practice for federally-funded multisite programs. However, there is a tendency for site-level projects to focus just on their reporting responsibilities and overlook the potential use of data in informing local programs. This paper will present the lessons learned from Hawaiis SPF-SIG (Strategic Prevention Framework State Incentive Grant) Project in its participation in SAMHSAs cross-site evaluation (2006-2012). The presenters will demonstrate the use of cross-site measurements in exploring program factors that influenced the outcomes of 13 prevention education programs implemented in Hawaii. Multilevel modeling technique was applied to examine various drinking indicators of youth participants 12-17 years old at program completion. The selected cross-site measurements were able to explain 24%-59% of the variances observed at the program level. Finally, the presenters will discuss the pathway from evaluation findings to program enhancement at the local level and how local evaluation could strengthen future cross-site evaluation efforts.
Poster 3: Testing for Sustainability: What's Left Five Years Later? Experiences and Results From Post-Intervention Evaluations (PIE)
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Kang Yue, Project Concern International (PCI), kyue@pciglobal.org
Emilia Rodriguez-Stein, Inter-American Foundation, erstein@iaf.gov
Nanette Barkey, Plan International, nanette.barkey@planusa.org
Abstract: For the past 20 years, sustainability has been a key concept in development debates and field programs. Yet, sustainability itself is not a well understood, defined or measured concept. Thus, sustainability of program results is rarely tested. Final evaluations often focus on the results created by the project and say little about the likelihood of sustainability or the strategies and interventions that may lead to it. This workshop presents the experiences of three organizations that have conducted a total of 12 "Post-Intervention" evaluations five or more years after project close-out - attempting to better understand sustainability, define it and identify components of the concept.
Poster 126: A Comprehensive Multi-level Evaluation of an Academic Support Program for College Students With Learning Differences
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Guili Zhang, East Carolina University, zhangg@ecu.edu
Sarah Williams, East Carolina University, williamssar@ecu.edu
Diane Majewski, East Carolina University, majewskid@ecu.edu
Abstract: To provide access to higher education and to foster academic success for students with learning differences, the College STAR (Supporting Transition, Access, and Retention) initiatives were designed and implemented at a public research university in the southeastern United States. The College STAR program includes both a student support component (Component A) and an instructional support component (Component B). As such, the program involves multiple stake holders and generates intended main effects as well as unanticipated spillover impacts. We designed and conducted a comprehensive evaluation at three levels to assess the program's implementation and impacts: student, faculty, and institution. This proposal provides an overview of the evaluation design and evaluation findings. In particular, unique challenges arise from evaluating such a program with complex structure are discussed, and effective strategies are introduced.
Poster 235: Does Self-employment Improve Seniors' Health? The Implication of Health Impact of Self-employment
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Ting Zhang, University of Baltimore, tzhang@ubalt.edu
Dawn Carr, Stanford University, carrdc@stanford.edu
Abstract: This paper examines the health impact of being self-employed among seniors (aged 62+) and its implications. Facing an aging workforce, self-employment at older ages is not only an important economic issue, it is also related to seniors' mental and physical wellbeing. Relying on the latest 6 waves data from the Health and Retirement Study (HRS), the study adopts a frailty index to measure seniors' comprehensive health and conducts a 2 stage generalized panel data model with instrument variables and incorporating spatial effects. Instrument variable estimators measuring work related health limitations are adopted to control potential endogeneity and simultaneity issues. In addition to the impact of self-employment and age on seniors' health, spatial effect is found to be important. The self-employment impact on health is addressed in the context of age, industry, and other social and personal factors. The paper concludes with research findings, implications, and future research directions.
Poster 115: Practices and Methods of Institutes' Expert Assessment in Chinese Academy of Sciences
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Jian-zhong Zhou, Chinese Academy of Sciences, jzzhou@casipm.ac.cn
Xiao-xuan LI, Chinese Academy of Sciences, xiaoxuan@casipm.ac.cn
Fang Xu, Chinese Academy of Sciences, xufang@casipm.ac.cn
Bing Shi, Chinese Academy of Sciences, 
Tao Dai, Chinese Academy of Sciences, daitao@casipm.ac.cn
Guoliang Yang, Chinese Academy of Sciences, glyang@casipm.ac.cn
Abstract: Quantitative data such as papers, patents was adopted for monitoring research institutes in Chinese Academy of Sciences. Nowadays with the development of innovation capability and changes in external environment, quantitative method is no longer able to meet the development needs of the Institutes. Therefore, in 2011 CAS proposed to invite high level experts for diagnostic assessment of research institutes. However CAS is a large research institute with various types of affiliated institutes crossing many disciplines, how to undertake expert diagnostic assessment in such complicated system is worth considering. Last year CAS selected four representative institutes as a prior study of expert diagnostic assessment. Based on the different characteristics of selected institutes, four models of expert diagnostic assessments are identified from the practice which is helpful for the future.
Poster 102: An Excel-based System for Analysis of Smaller Text-based Data Sets
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Yifei Zhu, Florida State University, yz11@my.fsu.edu
Abstract: It is often more efficient and cost-effective to analyze smaller qualitative data sets with software that most researchers have already installed on their computers. In this poster session, I will show how I conducted the qualitative analysis for the NanoCORE II curriculum project at the Florida Agricultural and Mechanical University/Florida State University (FAMU/FSU) College of Engineering. In the poster presentation I will focus on the three-step method I used for bringing raw text data into a qualitative report: - combing data for themes, using excel - selecting quotes to support themes - writing the technical report The contribution of the poster session is to present a system for analyzing small text-based data sets without the cost or investment in powerful and complex qualitative data analysis software.
Poster 190: Measurement Invariance for the LibQUAL+™ Instrument Across Measurement Occasions
Poster Presentation 101 to be held in International Center on Wednesday, Oct 16, 7:00 PM to 8:30 PM
Presenter(s):
Tekeisha Zimmerman, University of North Texas, zimmermantk@sbcglobal.net
Kim Nimon, University of North Texas, kim.nimon@unt.edu
Abstract: The LibQUAL+™ is a web-based self report instrument used to assess and improve library services. This 22 item survey measures users' perceptions of quality across three constructs: Affect of Service, Information and Control, and Library Place. Although the survey measures these constructs across three occasions (minimum service level, desired service level, perceived service performance), no studies establishing invariance in this area have been published. To address this gap, this study will test measurement invariance for the LibQUAL+™ across measurement occasion using data collected from university students over a six- year period from 2005 until 2011. Specifically, data from four samples collected at two-year intervals will be analyzed. Confirmatory analysis methods will be used to test invariance and stability of the three factor structure of the LibQUAL+™.

Return to Evaluation 2013 Home Page
Search Results for All Sessions