2014 INNOVATION AND ACCOUNTABILITY IN LANGUAGE PROGRAM EVALUATION

Permanent URI for this collection

Browse

Recent Submissions

Now showing 1 - 10 of 12
  • Item
    Reframing assessment: Innovation and accountability between the global and the local
    (Heinle Cengage Learning, 2014-01-01) Cachey Jr., Theodore J.
    This chapter highlights the roles and impact of assessment in a large multilingual department, from the experience and perspective of the chairperson. It argues for the value of useful assessment as a means of reframing the goals of language and culture programs and suggests that a more robust assessment culture might help languages and literatures to achieve the kind of reform that the Modern Languages Association called for in its widely discussed 2007 report “Foreign Languages and Higher Education: New Structures for a Changed World.” The approach to assessment outlined here aims to be useful, sustainable, and not too burdensome for faculty members. How the assessment project was started, how it has focused on a selected outcome in each assessment cycle, what kind of tools have been used to collect data, and the ways in which the assessment process can lead to curricular adjustments are also described. The project has contributed to building mutual respect and collegiality across the lecturer and research faculty frontier; has proven to be an incubator of curricular innovation; and has helped faculty members, both individually and collectively, to become more effective advocates for the importance of the languages and literatures other than English within the humanities. Finally, the chapter argues that engaging in assessment requires engaging with the messy world of higher education as it is and not as we wish it might be.
  • Item
    Do we speak the same language?: The iterative development of an institutionally mandated foreign language assessment program
    (Heinle Cengage Learning, 2014-01-01) Myers, Lindsy ; Lindsay, Nathan
    This chapter describes the development and implementation of a dynamic and resourceful assessment program within a Foreign Language Department in response to a university-wide mandate. It outlines the assessment struggles and successes of a program from its inception, and explores the relationship between university requirements and the Foreign Language Department’s needs, skills, and goals. In particular, the chapter presents the policies, practices, and structures involved in this longitudinal project. Both positive and negative effects of a university template are outlined, including how the department is attempting to find its own voice and meaning in the process.
  • Item
    Foreign language course grades as prerequisites and programmatic game keepers
    (Heinle Cengage Learning, 2014-01-01) Brown, Alan Victor
    Stakeholders within and beyond the academy are eager to know to what extent FL programs are responsible for producing valuable student-learning outcomes that reflect not only curriculum-specific achievement but also classroom-independent, real-world proficiency. At the heart of the issue resides the ambiguous, nondescript, and somewhat idiosyncratic nature of course grades. This chapter reports, first, on a descriptive analysis of grade-based prerequisites as they appear in electronic undergraduate Spanish program catalogues at all 73 research-intensive public institutions, as classified by the Carnegie Foundation. The quantitative analysis includes descriptive statistics on the use of grade-based metrics as prerequisites for undergraduate courses, that is, previous course grade and grade point average (GPA). The qualitative analysis examined the language and logic used in specifying these prerequisites. Following this analysis, a correlational study of one of the 73 programs is presented that explores the relationship between students’ grades from all sections of two mandatory pre-major courses and their performance on an external measure of reading, speaking, and writing proficiency. Within this program, students must earn an A or a B in both courses before continuing with the Spanish major, so grades carry high-stakes consequences and serve as programmatic gatekeepers. The chapter concludes with general recommendations on how to sensibly approach program articulation, course prerequisites, and the nature and composition of university-level FL course grades.
  • Item
    Promoting change in the “two-tiered” departments: An exploratory evaluation of conflict and empowerment levels among language and literature faculty
    (Heinle Cengage Learning, 2014-01-01) Zannirato, Alessandro
    This chapter documents an exploratory evaluation, carried out at the national level on the organizational effects of the “two-tiered” system in departments of Foreign Languages and Literatures (FLL) in the United States. The author initiated and conducted the evaluation for the purposes of (a) gaining a better understanding of the organizational dynamics that may arise in two-tiered departments; (b) exploring solutions to the identified challenges; and (c) identifying a community of language and literature professionals in both first-tier and second-tier positions interested in addressing the issues raised by the MLA report in a more inclusive way. After a theoretical comparison of the two-tiered system with similar configurations described in the literature, the results of an exploratory study concentrating on the level of organizational conflict and empowerment in FLL departments will be presented. The findings suggest that conflict and disempowerment may be factors affecting the interactions between language and literature faculty. The users will be able to draw on the findings of this evaluation to (a) reflect on their local reality and identify structural factors that may be conflict generating and disempowering; (b) use the empowerment strategies suggested in this chapter as a starting point to devise their own strategies in order to reduce inequities in FLL departments; and (c) engage in further discussion with the evaluator, provide further input, discuss the creation of a working group equally representing first-tier and second-tier faculty, and collectively prioritize a plan of action.
  • Item
    From frameworks to oversight: Components to improving foreign language program efficacy
    (Heinle Cengage Learning, 2014-01-01) Klee, Carol ; Melin, Charlotte ; Soneson, Dan
    This chapter advocates the use of an aspirational framework for student-learning outcomes that aligns the intellectual content of general education objectives with the values expressed in the ACTFL National Standards for language study. Such an approach emphasizes the benefits to language programs that can be realized through clarity in expectations and the systematic, longitudinal evaluation of outcomes. It also has far-reaching policy implications for foreign language programs, which the authors explore. The chapter introduces and discusses a rubric that models the connections between linguistic and intellectual skills. In addition, outcomes benchmarks for most commonly taught languages and less commonly taught language groups are proposed. The involvement of faculty SLA experts (tenured or tenure-track) is seen as key to the process of effective curricular reform, which must also involve other faculty members and stakeholders in active collaboration to implement system-wide evaluation practices and ongoing curricular reform.
  • Item
    Seat time versus proficiency: Assessment of language development in undergraduate students
    (Heinle Cengage Learning, 2014-01-01) Shmitt, Elena
    This chapter reports the results of a longitudinal study of language proficiency after three semesters of the required study of foreign languages at an urban university in the United States. The study was designed with two goals: (1) to better understand the implications of a transition from a seat-time requirement to a proficiency requirement and (2) to determine how scheduling might affect the efficacy of language learning. The study was motivated by a university-wide movement toward a new proficiency-driven Liberal Education Program (LEP). The LEP requires that students demonstrate Intermediate Low proficiency on the ACTFL proficiency scale in the language of their choice as measured by the Standards- Based Measurement of Proficiency (STAMP) test. A three-year study was carried out to determine the feasibility of this requirement. Specifically, it aimed to (a) establish the percentage of students who possess all the skills tested by STAMP at the Intermediate Low level after three semesters of language study; (b) identify areas of weakness as demonstrated by the test; and (c) determine the most effective schedule of classes.
  • Item
    Designing an embedded outcomes assessment for Spanish majors: Literary interpretation and analysis
    (Heinle Cengage Learning, 2014-01-01) Liskin-Gasparro, Judith E. ; Vasseur, Raychel
    The assessment of student learning outcomes comprises two stages: designing and conducting the assessment and making use of the results of the assessment to improve the program. This chapter deals with the first stage: designing and implementing an assessment of the knowledge and skills in literary interpretation and analysis of graduating Spanish majors at the University of Iowa. We begin with background on the project and the student learning outcomes for the Spanish. We describe the process, grounded in Patton’s (2008) utilization-focused evaluation framework, of developing the assessment rubric, and we then report on its use in an operational assessment. In the discussion, we consider the impact of the assessment on the curriculum in the Spanish major and implications of this project for language program directors.
  • Item
    Student analytics and longitudinal evaluation of language programs
    (Heinle Cengage Learning, 2014-01-01) Ecke, Peter ; Ganz, Alexander
    This chapter provides strategies and examples of how program administrators can monitor various important indicators and utilize technological tools and resources for the collection and analysis of program-relevant data. The chapter identifies and addresses four challenges of language program evaluation: (1) the long-term tracking, monitoring, and interpretation of student enrollments, taking into account both internal (departmental and institutional) and external (regional and national) data; (2) the identification of potential issues of the program and subsequent intervention; (3) the establishment of student profiles relevant for program planning and design; and (4) the tracking of individual students’ success and special achievements in their career paths. The chapter makes a case for the use of internal student surveys, complemented by internal and external data, as essential components of a continuous, internally driven program evaluation that may be useful for periodic academic program reviews.
  • Item
    The development, management, and costs of a large-scale foreign language assessment program
    (Heinle Cengage Learning, 2014-01-01) Bernhardt, Elizabeth ; Brilliantes, Monica
    The Stanford Language Center has a 20-year history of designing, implementing, and managing an assessment program across two years of instruction in 15 languages. This assessment program is embedded in the teaching, learning, and professional development facets of the Language Center. Aspects of the assessment program have been chronicled over the years. This chapter reiterates the scope of the assessment program (number of students assessed each year); the assessment measures (Simulated Oral Proficiency Interviews [SOPIs]; Oral Proficiency Interviews [OPIs]; Writing Proficiency Assessments [WPAs]; and Writing Proficiency Tests [WPTs]); the relationship of these measures to the curricular objectives and outcomes based in the National Standards; and the impact the assessment program has had on the status and authority of the Language Center within the university, particularly with regard to the university’s accreditation process. The bulk of the chapter focuses on the monetary and nonmonetary costs of this program. The nonmonetary costs in a large assessment program revolve around both teaching staff and students. The teaching staff has to be on the same page regarding the scoring schemes and scales for any assessment. Students, too, must be convinced that any assessment program is really about improving their performance as well as that of their peers. Monetary costs include test development, technology costs for realizing the tests in digital format, teacher training costs, and the costs of rating the assessments. Finally, and perhaps most importantly, the chapter provides information on the relationship between the monetary investment in the assessment program and student performance.
  • Item
    Evaluation capacity building in college language programs: Developing and sustaining a student exit survey project
    (Heinle Cengage Learning, 2014-01-01) Kondo-Brown, Kimi ; Davis, John McE. ; Watanabe, Yukiko
    Faculty attitudes toward institutionally mandated (or accountability driven) program evaluation demands vary along a continuum of proactive to reactive stances. Such variation is partly related to how individual faculty members perceive the ultimate users and uses of evaluation, as well as the workload associated with evaluation activities. Thus, in order to conduct externally mandated program evaluation successfully, the institution needs to balance the evaluation needs of various stakeholders—both within and beyond departments and programs—and invest in evaluation capacity building that supports faculty evaluation efforts. The purpose of this chapter is to describe the processes and outcomes of a college-level program evaluation initiative, as well as illuminate key issues and challenges in postsecondary program evaluation. Specifically, we discuss how the dean’s office of the College of Languages, Linguistics, and Literature (CLLL) at the University of Hawai‘i at Ma-noa built evaluation capacity to develop and sustain an online student exit survey system. Despite challenges with survey administration and use of data, the initiative has had a number of meaningful, productive outcomes for CLLL faculty. The general impact of the project, we feel, has been an increased capability on the part of CLLL faculty and staff to make evidence-based decisions about program development. This chapter discusses the lessons the college has learned from the exit survey evaluation initiative and makes suggestions for other institutions planning to undertake similar evaluation projects.