| JPAM Preview ▪ February 2014 |
| JPAM Preview is a newsletter that calls attention to forthcoming articles in JPAM. JPAM Preview provides brief summaries of content now available digitally in Early View, Wiley's online publication system. |
| |
| |
| Feature Article The Use and Efficacy of Capacity-Building Assistance for Low-Performing Districts: The Case of California's District Assistance and Intervention Teams Katharine O. Strunk, Andrew McEachin, and Theresa N. Westover The theory of action upon which high-stakes accountability policies are based calls for systemic reforms in educational systems that will emerge by pairing incentives for improvement with extensive and targeted technical assistance (TA) to build the capacity of low-performing schools and districts. To this end, a little discussed and often overlooked aspect of the No Child Left Behind Act (NCLB) mandated that, in addition to sanctions, states were required to provide TA to build the capacity of struggling schools and Local Education Agencies (LEAs, or districts) to help them improve student achievement. Although every state in the country provides some form of TA to its lowest performing districts, we know little about the content of these programs or about their efficacy in improving student performance. In this paper, we use both quantitative and qualitative analyses to explore the actions taken by TA providers in one state-California-and examine whether the TA and support tied to California's NCLB sanctions succeeds in improving student achievement. Like many other states, California requires that districts labeled as persistently failing under NCLB (in Program Improvement year 3, PI3) work with external experts to help them build the capacity to make reforms that will improve student achievement. California's lowest performing PI3 districts are given substantial amounts of funding and are required to contract with state-approved District Assistance and Intervention Teams (DAITs), whereas the remaining PI3 districts receive less funding and are asked to access less intensive TA from non-DAIT providers. We use a five-year panel difference-in-difference design to estimate the impacts of DAITs on student performance on the math and English language arts (ELA) standardized tests relative to non-DAIT TA during the two years of the program intervention. We find that students in districts with DAITs perform significantly better on math California Standards Tests (CSTs) averaged over both treatment years and in each of the first and second years. We do not find evidence that students in districts with DAITs perform higher on ELA CSTs over the combined two years of treatment, although we find suggestive evidence that ELA performance increases in the second year of treatment relative to students in districts with non-DAIT TA. Ordinary least squares (OLS) regressions that explore the association between specific activities fostered by DAITs and changes in districts' gains in achievement over the two years of treatment show that DAIT districts that report increasing their focus on using data to guide instruction, shifting district culture to generate and maintain high expectations of students and staff, and increasing within-district accountability for student performance, have higher math achievement gains over the course of the DAIT treatment. In addition, DAIT districts that increase their focus on ELA instruction and shift district culture to one of high expectations have higher ELA achievement gains than do DAIT districts that do not have a similar focus. © 2012 by the Association for Public Policy Analysis and Management. Forthcoming in JPAM 33(3). Link to JPAM Early View. If you want to cite this article before it is in print, please use the DOI number listed with each article. |
| |
| Feature Article Revisiting the Income Tax Effects of Legalizing Same-Sex Marriages James Alm, J. Sebastian Leguizamon and Susane Leguizamon In this paper we estimate the impacts on income tax collections of legalizing same-sex marriage. We utilize new individual-level data sources to estimate the federal income tax consequences of legalizing same-sex marriages. These data sources also allow us to estimate the impact of legalization on state income tax collections. We find that 23 states would realize a net fiscal benefit from legalization, while 21 states would experience a decline in revenue. The potential (annual) changes in state tax revenue range from negative $29 million in California to positive $16 million in New York. At the federal level, our estimates suggest an overall reduction in revenues, ranging from a potential loss of $187 million to $580 million. Overall, we find that the federal and state impacts are quite modest. We also find that our estimates are only marginally affected by alternative assumptions about how many same-sex couples will choose to marry and which partner will claim any children for tax deduction purposes. Forthcoming in JPAM 33(2). Link to JPAM Early View. If you want to cite this article before it is in print, please use the DOI number listed with each article. |
| |
| Feature Article The Forgotten Summer: Does the Offer of College Counseling after High School Mitigate Summer Melt Among College-Intending, Low-Income High School Graduates? Benjamin L. Castleman, Lindsay C. Page, and Korynn Schooley Despite decades of policy intervention to increase college entry and success among low-income students, considerable gaps by socioeconomic status remain. To date, policymakers have overlooked the summer after high school as an important time period in students' transition to college, yet recent research documents high rates of summer attrition from the college pipeline among college-intending high school graduates, a phenomenon we refer to as "summer melt." We report on two randomized trials investigating efforts to mitigate summer melt. Offering college-intending graduates two to three hours of summer support increased enrollment by 3 percentage points overall, and by 8 to 12 percentage points among low-income students, at a cost of $100 to $200 per student. Further, summer support has lasting impacts on persistence several semesters into college. Forthcoming in JPAM 33(2). Link to JPAM Early View. If you want to cite this article before it is in print, please use the DOI number listed with each article. |
| |
| Special Symposium on Qualitative and Mixed-Methods for Policy Analysis Katharine Edin, Guest Editor Maureen A. Pirog Qualitative and Mixed Methods This symposium, devoted to qualitative and mixed-method research, reflects our view that such approaches are vital tools for policy research. A growing body of research demonstrating the utility of such approaches has appeared in the pages of social science journals. Qualitative and mixed methods have not been well-represented in JPAM, an omission we hope that this symposium will begin to correct. In the past, JPAM has received very few submissions using rigorous qualitative or mixed-methods papers, and when received, referees have not reacted kindly. The announcement of this symposium was intended to signal a willingness on the part of the editor to publish such studies, and the selection of a scholar known for qualitative and mixed methods as a guest editor was additionally intended to convey the message that referees open to such approaches would be selected to review the papers. We see three distinct virtues of qualitative and mixed-methods approaches. In this introduction, we illustrate these virtues by providing some examples of extant policy research. Many of these examples are drawn from research on housing, per the expertise of the guest editor. Along with the extant literature, the articles we have selected for the symposium are also meant to showcase these virtues. Forthcoming in JPAM 33(2). Link to JPAM Early View. If you want to cite this article before it is in print, please use the DOI number listed with each article. |
| |
| Special Symposium on Qualitative and Mixed-Methods for Policy Analysis Partnering and Parenting in Poverty: A Qualitative Analysis of a Relationship Skills Program for Low-Income Unmarried Families Jennifer M. Randles Since the mid-1990s, the federal government has funded numerous relationship skills programs, including some specifically targeting low-income, unmarried parents, in an effort to strengthen couples' relationships and increase family stability. The previous research on the effectiveness of these interventions has revealed mixed results about whether such programs can improve the relationships of lower income couples who tend to experience lower relationship quality, lower marriage rates, and higher rates of relationship dissolution. This article draws on in-depth qualitative data collected during an 18-month ethnographic study of one federally funded relationship skills program for unmarried, low-income couples expecting a new baby. Overall, though parents found the financial management lessons included in the classes only minimally useful, if at all, they found other aspects of the program particularly useful for three main reasons: (1) classes allowed parents to focus exclusively on their couple relationships in ways they rarely did otherwise; (2) program incentives helped parents make financial ends meet that month; and (3) parents learned that the challenges they personally experienced were often endemic to the romantic and co-parenting relationships of unmarried parents who have few resources and experience more challenges that tend to undermine relationship quality, such as financial stress and relational ambiguity. Engaging with other couples around shared challenges normalized couples' relationship problems and lessened the resentment and animosity that typically characterized their partner interactions. These findings have important implications for healthy marriage and relationship policy. Program developers should avoid lessons that imply low-income, unmarried parents' spending habits and family-formation decisions are deficient. Interventions should instead encourage couples to discuss their shared challenges and minimize their tendency to individualize relational and financial strain. Forthcoming in JPAM 33(2). Link to JPAM Early View. If you want to cite this article before it is in print, please use the DOI number listed with each article. |
| |
| Special Symposium on Qualitative and Mixed-Methods for Policy Analysis Tax Code Knowledge & Behavioral Responses Among EITC Recipients: Policy Insights for Low-Income Unmarried Families Laura Tach and Sarah Halpern-Meekin We build on the robust quantitative literature on behavioral responses to the Earned Income Tax Credit (EITC) by using in-depth qualitative interviews with 115 EITC recipients to examine how they understand and respond to its incentive structures regarding earnings, marriage, and childbearing. We find that respondents consider their tax refund as a whole, without differentiating the portion from the EITC; as a result, they cannot predict how their EITC refund would change if they altered their labor supply or marital status. Incentives for childbearing are better understood, but are not specific to the EITC; rather, parents respond to a combination of tax deductions and credits as a whole. Respondents would like to maximize their refunds, but most cannot or would not alter their behavior due to structural constraints they face in the labor and marriage markets. Rather than adjust work hours, defer marriage, or have additional children, respondents exhibit a different type of behavioral response to the incentive structure of the EITC: They alter their tax filing status in order to maximize their refunds. They routinely claim zero exemptions and deductions on their W-4s, file their tax returns as head of household rather than as married, and divide children among the tax returns of multiple caregivers. Although some of these behaviors qualify as tax noncompliance, they emerge because the intricacies of the tax code conflict with the complexity and fluidity of finances and family life in low-income households. Forthcoming in JPAM 33(2). Link to JPAM Early View. If you want to cite this article before it is in print, please use the DOI number listed with each article. |
| |
| Special Symposium on Qualitative and Mixed-Methods for Policy Analysis Making Ends Meet after Prison David J. Harding, Jessica J.B. Wyse, Cheyney Dobson, and Jeffrey D. Morenoff Former prisoners are at high risk of economic insecurity due to the challenges they face in finding employment and to the difficulties of securing and maintaining public assistance while incarcerated. This study examines the processes through which former prisoners attain economic security, examining how they meet basic material needs and achieve upward mobility over time. It draws on unique qualitative data from in-depth, unstructured interviews with a sample of former prisoners followed over a two- to three-year period to assess how subjects draw upon a combination of employment, social supports, and public benefits to make ends meet. Findings reveal considerable struggle among our subjects to meet even minimal needs for shelter and food, although economic security and stability could be attained when employment or public benefits were coupled with familial social support. Sustained economic security was rarely achieved absent either strong social support or access to long-term public benefits. However, a select few were able to leverage material support and social networks into trajectories of upward mobility and economic independence. Policy implications are discussed. Forthcoming in JPAM 33(2). Link to JPAM Early View. If you want to cite this article before it is in print, please use the DOI number listed with each article. |
| |
| Special Symposium on Qualitative and Mixed-Methods for Policy Analysis Improving the Implementation and Effectiveness of Out-of-School-Time Tutoring Carolyn J. Heinrich, Patricia Burch, Annalee Good, Rudy Acosta, Huiping Cheng, Marcus Dillender, Christi Kirshbaum, Hiren Nisar, and Mary Stewart School districts are spending millions on tutoring outside regular school day hours for economically and academically disadvantaged students in need of extra academic assistance. Under No Child Left Behind (NCLB), parents of children in persistently low-performing schools were allowed to choose their child's tutoring provider, and together with school districts, they were also primarily responsible for holding providers in the private market accountable for performance. We present results from a multisite, mixed-method longitudinal study of the impact of out-of-school time (OST) tutoring on student reading and mathematics achievement that link provider attributes and policy and program administration variables to tutoring program effectiveness. We find that many students are not getting enough hours of high-quality, differentiated instruction to produce significant gains in their learning, in part because of high hourly rates charged by providers for tutoring. We identify strategies and policy levers that school districts can use to improve OST tutoring policy design and launch improved programs as waivers from NCLB are granted. Forthcoming in JPAM 33(2). Link to JPAM Early View. If you want to cite this article before it is in print, please use the DOI number listed with each article. |
| |
| Methods for Policy Analysis When Do Regression-Adjusted Performance Measures Track Longer-Term Program Impacts: A Case Study for Job Corps Peter Z. Schochet and Jane Fortson The use of performance management systems has increased since the Government Performance and Results Act of 1993. While these systems share the goal of trying to improve service delivery and participant outcomes, they do not necessarily provide information on the causal (value-added) effects of a program, which requires a rigorous impact evaluation. One approach for potentially improving the association between program performance measures and impacts is to adjust performance measures for differences across performance units in participant characteristics and local economic conditions. This article develops a statistical model that describes the conditions under which regression adjustment improves the performance–impact correlation. We then use the model to examine the performance–impact association using extensive data from a large-scale random assignment evaluation of Job Corps, the nation's largest training program for disadvantaged youths. We find that while regression adjustment changes the Job Corps center performance measures, the adjusted performance measures are not correlated with the impact estimates. The main reasons are the weak associations between the unadjusted Job Corps performance measures and participants' longer-term outcomes as measured by the evaluation, as well as the likely presence of unobserved factors across centers that are correlated with outcomes. Forthcoming in JPAM 33(2). Link to JPAM Early View. If you want to cite this article before it is in print, please use the DOI number listed with each article. |
| |
| Point/Counterpoint Theory, Methods, and Data Kenneth A. Couch, Editor As conceptual approaches to policy analysis evolve, both research methods and data requirements are impacted. Similarly, new fields of inquiry are opened as data availability advances. In past decades, conceptual insights regarding sample selection and the validity of estimated impacts from a given sample have driven important methodological advancements such as the two-step correction method, fixed effects, regression discontinuity, and propensity score matching. The increased availability of agency-level data and national collations of administrative records matched to survey information has allowed researchers to examine topics that previously would not have been possible. Here, I have asked two leading scholars with a deep understanding of the evolution of policy analysis and current trends to discuss developments that are likely to be important to the field in the future. Participating in the discussion are Thomas Cook and Maureen Pirog. Thomas Cook holds the Joan and Serepta Harrison Chair of Ethics and Justice and is a Professor of Sociology, Psychology, and Social Policy at Northwestern University. Maureen Pirog is Rudy Professor of Public and Environmental Affairs at the School of Public and Environmental Affairs, Indiana University, and Affiliate Faculty at the Evans School of Public Affairs, University of Washington. Maureen Pirog is also the Editor-in-Chief of the Journal of Policy Analysis and Management. To frame the discussion, I asked the authors to address the following questions: 1. What are the likely key advances in theory, analytical methods, and data that are and will likely continue to reshape public policy and management research over the next decade? 2. What are the ramifications of these breakthroughs, advances, or trends for the actual conduct of public policy and management research? Forthcoming in JPAM 33(2). Link to JPAM Early View. If you want to cite this article before it is in print, please use the DOI number listed with each article. |
| |
| Point/Counterpoint Generalizing Causal Knowledge in the Policy Sciences: External Validity as a Task of Both Multiattribute Representation and Multiattribute Extrapolation Thomas D. Cook I have been asked to write about methodological issues likely to be prominent in future public policy research. Many issues would deserve attention in a longer presentation, but here I want to concentrate on external validity and its links to evidence-based policy. Such policy uses social science knowledge about what has worked in the past to inform policy decisions in the future. This requires justified procedures for describing the populations of persons, settings, and times in which a given causal relationship has been demonstrated to date, and justified procedures for moving from operational details about the cause and effect to the category labels we use to designate the more general cause or effect constructs. We call this the representation function since the need is to know what the sampling particulars represent as more general populations or categories. The traditional sampling theory framing of this issue would be the following: Given the populations of persons, settings, times, and treatment and outcome constructs to which I want to generalize, how well do the specifics actually sampled match these populations or categories? Forthcoming in JPAM 33(2). Link to JPAM Early View. If you want to cite this article before it is in print, please use the DOI number listed with each article. |
| |
| Point/Counterpoint Data Will Drive Innovation in Public Policy and Management Research in the Next Decade Maureen A. Pirog While there are many strong forces at play in the field of policy analysis and management, the strongest and most fundamental changes that will shape the field of policy analysis over the next decade are in the types of data that are increasingly available to researchers. Because research designs and statistical/econometric approaches are codependent with data, changes in the types of data available to public policy and management researchers will transform the field. Additionally, continuing research on the ability of econometrics to compensate for selection biases will continue to push researchers interested in the does it work question further down the path toward random assignment studies. Creative work on better quasi-experimental methods could change this movement, but in the absence of better econometrics, researchers will continue the movement into experimental research. Let me begin with the low-lying fruit, changes in the types and accessibility of data. Forthcoming in JPAM 33(2). Link to JPAM Early View. If you want to cite this article before it is in print, please use the DOI number listed with each article. |
| |
| Point/Counterpoint "Big Data" in Research on Social Policy Thomas D. Cook Before embarking on a counterpoint to Professor Pirog's presentation, I want to thank her for her many years as editor of JPAM. She achieved great improvements in article quality, breadth of topic coverage, and efficiency of management. All JPAM readers are in her debt. I have space here to comment on only one of her themes, but a theme important enough to reoccur in several plenary-type addresses at the 2013 APPAM annual convention: How "big data" might affect research in the policy sciences. In my view, the influences will be almost all positive, but they will be larger in some policy areas than others. The benefits of big data speak more to improving description and prediction than to improving the causal and explanatory knowledge that are so critical to both science and policy. This is because substantive theory and experimental design are particularly important for causation and explanation, but are not obvious consequences of big data. While some improvements in the areas of causation and explanation are to be expected in ways we describe, we doubt that they will be larger than what big data achieves for description and prediction. Forthcoming in JPAM 33(2). Link to JPAM Early View. If you want to cite this article before it is in print, please use the DOI number listed with each article. |
| |
| Point/Counterpoint Internal versus External Validity: Where Are Policy Analysts Going? Maureen A. Pirog First, I want to thank Professor Cook for his many contributions to the field of policy analysis. I have a deep regard for him as evidenced by my many dog-eared, underlined, and highlighted copies of his books. To engage in this discussion with him is an honor. And to validate his discussion of casual inference and extrapolation from utosti to UTOSTi to *UTOSTi, I admit that I wrote my response to his opening statement at the Wagner School of Public Service, and then promptly lost the flash drive with it en route to Indiana University and then the National Institute for Development Administration (NIDA) in Thailand. So sitting at NIDA, reading my second response to his opening statement, I see no similarities between my first response to his opening statement and the second-demonstrating that a change in time (ti) and setting (s) has resulted in a different outcome (o). Presumably I (u) have not changed a great deal in less than a month and neither has his opening statement (t). External validity has indeed taken a backseat to internal validity in policy research. When we have difficulty making causal inferences from utosti to UTOSTi, the leap to *UTOSTi seems formidable indeed. Nonetheless, policymakers, in the absence of evidence of how treatments work for different types of people or units (u), in different settings (s), and time periods (ti), must either implement new programs or continue with the status quo. In reality, evidence-based policymaking means taking research results or evidence about UTOSTi and assuming that these findings will probably be about the same for different units, treatments or treatment dosages, outcomes, settings, and time periods. We all recognize that this is an exceptionally strong assumption. However, if it is more reasonable to assume that rigorous findings about the results of some treatment on a specific set of outcomes will not likely be replicated for other units, settings, and time periods, then we might need to resign as policy analysts, just when we were becoming nerdy, geeky, wonky sexy! The tension is the trade-off between the practical timelines of policymakers and the desire for strong causal inference from utosti to UTOSTi and then extrapolation to *UTOSTi. Forthcoming in JPAM 33(2). Link to JPAM Early View. If you want to cite this article before it is in print, please use the DOI number listed with each article. |
| |
| Journal of Policy Analysis and Management is published by Wiley Periodicals on behalf of the Association for Public Policy Analysis and Management. Editor-in-Chief: Maureen Pirog ▪ Indiana University, School of Public and Environmental Affairs (and) University of Washington, Daniel J. Evans School of Public Affairs Managing Editors: Robert Kaestner ▪ University of Illinois at Chicago Christopher (Kitt) Carpenter ▪ Vanderbilt University | |
| For any comments or inquiries regarding JPAM, please contact us at jpam@indiana.edu. |
| |
| For Table of Contents (TOC) alerts, please follow the instructions below: 1. Create an account / sign in to Wiley Online Library. 2. Click on "Get New Content Alerts" in the JOURNAL TOOLS menu on the JPAM page. 3. An e-TOC shows up in your email inbox for each issue! |
| | | |
| | | | |