Transcription

1 Transforming Grades into Meaningful MPA Program Outcomes: Lessons from Benchmarking Learning Objectives in the Public Budgeting and Finance Course Esteban G. Dalehite Florida International University Abstract MPA programs have experienced increasing pressure from external stakeholders to produce indicators of program effectiveness. Producing these measures, however, can be both difficult and expensive. This article presents a low-cost method of transforming grades into meaningful MPA program outcomes. The method consists of (1) defining learning objectives, (2) structuring assessment instruments according to these learning objectives, and (3) using basic statistical procedures to determine whether a benchmark has been reached for each learning objective. The method presented in this article allows individual faculty to align course content with program mission, refine the specifics of program objectives, and measure the achievement of both. Hence the method is a good means of not only outcome assessment but also of faculty and curriculum development. Course specific indicators provide timely information on performance and can produce positive effects that spill over into the internship, capstone, and eventual realworld job experiences of students and alumni. The National Association of Schools of Public Affairs and Administration (NASPAA) and other accreditation bodies require that institutions of higher education periodically measure the outcomes of their master of public affairs or administration (MPA) programs. Most outcome measures that MPA programs use reflect external recognition, customer satisfaction, or a match between services supplied and demanded. For example, MPA programs often use information from JPAE 14(3): Journal of Public Affairs Education 413

2 the following sources to construct outcome measures: (1) admissions data on why students turned down the program, (2) student body demographic and scholastic aptitude data, (3) student entry surveys, (4) course evaluations, (5) internship supervisor surveys, (6) student exit surveys, (7) alumni surveys, (8) employer surveys, (9) enrollment, (10) faculty reviews, (11) faculty publication records, and (12) national rankings (Roberts and Pavlak, 2002a; Aristigueta and Gomes, 2006; Williams, 2002). Some of the outcomes built from these information sources fall under what Jennings (1989) calls the career success approach to conceptualizing outcomes. The focus of this article is a different source of information commonly offered by MPA programs to gauge achievement of program goals and objectives, namely grades and transcripts (Roberts and Pavlak, 2002a; NASPAA, 2004; Aristigueta and Gomes, 2006; Williams, 2002). Grades assess student learning of specific knowledge, skills, and values, and fall under what Jennings (1989) calls the value-added approach to conceptualizing outcomes. However, commonly reported grades reveal very little if anything about the specific knowledge, skills, and values that students learn and thus have little construct validity as measures of achievement of program mission and objectives. This article presents a method of transforming grades into meaningful MPA program outcomes and lessons derived from the application of this method to the introductory public budgeting and finance course. The method consists of (1) defining learning objectives, (2) structuring assessment instruments according to these learning objectives, and (3) using basic statistical procedures to determine whether a benchmark has been reached for each learning objective. This method allows individual faculty to align course content with program mission, refine the specifics of program objectives, and measure the achievement of both. Hence the method is a good means of outcome assessment and of faculty and curriculum development. Course-specific indicators provide timely information on performance and, insofar as this leads to improvements, these indicators can produce positive effects that spill over into the internship, capstone, and eventual real-world job experiences of students and alumni, and hence improve indicators of external recognition as well. The additional time required by faculty to carry out this method is minimal. Background The method presented in this article was applied to the core course in public budgeting and finance at Florida International University s (FIU) public administration program, during the spring of The master of public administration at FIU was established in It was first accredited by NASPAA in 1982 and has since been reaccredited in 1990, 1998, and The MPA program operates under an open admission policy. However, an average grade of B is required to graduate, and a probationary system is in place for students with low college 414 Journal of Public Affairs Education

3 GPAs. Student enrollment oscillates in the range. Most students are entry- or mid-level managers, and more than three-fourths have minority status. The method was developed in direct compliance with two internal policies: (1) a directive issued by the dean of the school for all faculty to incorporate learning goals and objectives in course syllabi; and (2) a resolution made by the program s faculty during the self-study year leading to the 2005 reaccreditation to define course-specific outcomes and measure them systematically. However, both of these policies exist in the context of the outcome assessment requirements placed on the program by NASPAA, the Southern Association of Colleges and Schools (FIU s regional accrediting body), and the Florida Board of Governors. It should be noted that discretion was left to individual professors to define and measure course-specific outcomes. Thus, the method presented here can be thought of as a bottom-up effort by individual faculty to produce course-specific program outcomes within the framework of a general program-wide requirement to do so. Nevertheless, the method is compatible with mission-curriculum alignment initiatives where central planning and monitoring are more prominent (Roberts and Pavlak, 2002a; Aristigueta and Gomes, 2006). Defining learning objectives The first step is to define learning goals or objectives. Learning goals are not new and have long been considered a staple of effective teaching. They provide professors and students a sense of direction, and when adequately conceptualized are indispensable for designing, implementing, and evaluating instruction (Lucas and Murry, 2002). Goal setting has long been considered a major mechanism to affect motivation, provided that goals are specific, challenging, and accepted (Latham and Locke, 1979). In addition to their pedagogical and motivational value, defining learning goals is equivalent to determining the specifics of program mission, course by course. Thus, it is an activity that provides individual faculty with some degree of discretion in defining the path of an academic organization, by identifying knowledge, skills, and values that are considered necessary for a successful career in public service. In addition, defining learning objectives provides the opportunity to take ownership of the course and to think critically and specifically through course materials. Course syllabi are often based on the structure of adopted texts or on syllabi designed by others. The public budgeting and finance course syllabus, prior to the application of this method, was no exception. One lesson learned from the application of the method is that this passive method of designing syllabi implies an uncritical acceptance of the explicit or implicit learning objectives of others. More technically, defining learning objectives and writing exam questions are instances of the processes of conceptualization and operationalization of constructs inherent in most research. The experience described in this article showed that this can easily be overlooked. Learning objectives represent the concepts or Journal of Public Affairs Education 415

4 constructs, and exam questions represent the operationalization of those constructs. Thus, defining learning objectives provides a benchmark against which the validity of the instruments of student learning assessment may be judged. Absent learning objectives, the validity standard tends to be vague, less articulated, and could probably be summarized as the entire content of textbook, supplemental readings, and assignments, plus perhaps topics the students should know. Such a broad learning goal makes for a diffuse program mission and casts doubt on the validity of assessment instruments. Several information sources were used to define learning objectives for the public budgeting and finance course. Chief among these were the program mission, the course catalog description, and NASPAA standards. These sources are reproduced in Table 1. As one can see, the FIU MPA program mission provided only a general orientation. The complement to the program mission, on the other hand, provided more specific guidance by stating that graduates should be well prepared for finance director or budget analyst positions. Naturally, the most detailed direction came from the catalog course description. It calls for a traditional course in public budgeting and finance similar to the overview course in budgeting required in most programs (Alexander, 1984). NASPAA standards help one realize that a course such as budgeting and finance does not necessarily have the sole purpose of teaching budgeting and financial skills. The course can and probably should encompass the broad areas of conveying knowledge (for example, public policy environment), developing critical thinking skills (for example, problem solving, analytical, policy or decision making skills), and understanding the values or the conflicts of values that may be at play. The public budgeting and finance course can also be used as a means of enhancing written and verbal communication skills through papers, projects, and presentations. In addition to the above mentioned sources, guidance for the definition of learning objectives was also derived from existing course materials, and student, peer, and stakeholder input. Other valuable sources were James R. Alexander (1984), Wes Clarke (2002) and Howard Frank (2002). Based on these inputs, general and specific learning goals or objectives were defined for the public budgeting and finance course, as seen in Table 2. These learning goals include fairly traditional materials that should be familiar to anyone who has taught the subject. They should not be seen as the perfect design for the public budgeting and finance course but rather as an iteration of a course that may continue to develop. This is one important lesson of the process presented in this paper. Defining learning objectives, if not done perfunctorily, is more the beginning of a journey than a one-shot activity. There is never a feeling of having fully finished the task, but rather the realization that learning objectives are likely to be defined and redefined overtime with experience, new insights, or new developments in the market for the educational services provided by MPA programs. In this sense, perhaps the best philosophy to have for the ongoing revi- 416 Journal of Public Affairs Education

5 Table 1. Guidance for Developing Learning Objectives Program mission found in the 2004 self-study report The Public Administration Program at Florida International University is a professional degree program providing a diverse student population with the knowledge, skills, and values for career preparation and advancement in public administration. The program places a special emphasis on the challenges confronting urban areas affected by internationalization (FIU-PAP, 2004, 2 3). Complement to the program mission found in the course catalog The Master of Public Administration prepares students for careers in public service and non-profit organizations. It also prepares students for private sector positions having significant contact with public organizations. The degree provides pre-service and mid-career students with an appropriate mix of technical and generalist skills needed for management and analytic positions in local, state, and federal government, as well as non-profit organizations. Students are given broad exposure to the field in the core and subsequently develop expertise within their areas of specialization. The degree also provides students with the necessary analytic and substantive background for successful pursuit of doctoral studies within the discipline. Graduates are well-prepared for positions as city manager, finance director, budget analyst, personnel director, special project coordinator and program analyst (FIU, 2005, 437). Course description found in catalog (FIU-PAP) PAD 6227 Public Finance and the Budgetary Process examines the theory and practice of public budgeting and its relationship to the administrative processes of control, management, and planning. Special emphasis will be given to the social balance question; the kinds and scope of government expenditures; the fiscal role of government in a mixed economy; sources of revenue available to government; administrative, political and institutional aspects of the budget and the budgetary process; and problems and trends in intergovernmental fiscal relations (FIU, 2005, 442) NASPAA standards 4.1, 4.2 and 4.21 (relevant aspects) 1. The purpose of the curriculum shall be to prepare students for professional leadership in public service. 2. The curriculum components are designed to produce professionals capable of intelligent, creative analysis and communication, and action in public service. 3. The curriculum components shall enhance the student s values, knowledge, and skills to act ethically and effectively: a. In the management of public service organizations, including budgeting and financial processes. b. In the application of quantitative and qualitative techniques of analysis, the components of which include: i. Policy and program formulation, implementation and evaluation, and ii. Decision-making and problem-solving c. With an understanding of the public policy and organizational environment, the components of which include: i. Political and legal institutions and processes. ii. Economic and social institutions and processes (NASPAA, 2006). Journal of Public Affairs Education 417

6 sion of learning objectives is the same that underlies zero-based budgeting. In other words, every year all learning objectives should be placed on the chopping block and justified as the most desirable set of knowledge, skills, and values for a successful career in public service. The learning objectives contained in Table 2 are classified by topic. However, it can be useful to also classify learning objectives according to the cognitive processes involved. Bloom s (1964) widely used typology distinguishes between the following stages in the progression of cognitive reasoning: knowledge, comprehension, application, analysis, synthesis, and evaluation. The latter three are categorized as higher-order skills. Considering this typology and the current design of the measuring instrument described below, learning objectives Table 2. Learning Objectives for the Core Course in Public Budgeting and Finance General learning objective The general learning objective can be titled budget and revenue competency and can be defined broadly as having the capacity to (1) develop, seek approval, and administer operating or capital budgets, (2) think critically and make decisions about budget process design and revenue systems, and (3) understand the history, politics, and theoretical underpinnings of major reform efforts and shifts in revenue structures across all levels of government in America. Specific learning objectives 1. Normative and positive foundations of public budgeting and finance (FPBF). The student should be sufficiently familiar with the rational paradigm, welfare economics, and the Thiebout hypothesis to have a good understanding of the efficiency gains and assumptions underlying normative prescriptions concerning (a) the role of the state in the economy, (b) budget reform, (c) the appropriate level of government to make tax-and-spend decisions, and (d) the design of revenue systems. In addition, the student should have a good understanding of the tradeoffs between efficiency and social equity as well as the roles of ideology and politics in determining tax-and-spend decisions. 2. The federal and state budget processes (Bud). The student should have a good understanding of the basic phases of a budget process, the federal budget process, the roles of its main players, and the basic differences between federal and state budget processes. This includes a good understanding of the advantages and disadvantages of executive-led versus legislative-led budget processes, and annual versus biannual budget processes. 3. Budget reform to rationalize spending decisions (PerfBud). The student should have a good understanding of federal and state reform efforts to rationalize spending decisions, the byproducts of these efforts, and an assessment of their success. This includes distinguishing between inputs, outputs, and outcomes, and their relationship with different budget formats. 4. Budget reform to control the deficit (Def). The student should have a good understanding of the drivers of and debate over excessive spending, the procedures established at the federal and state levels to control expenditures, and an assessment about whether they have worked. 418 Journal of Public Affairs Education

7 1 through 5 in Table 2 can be placed in a group of learning objectives in which the first three stages of Bloom s typology are being assessed. Learning objectives 9 and 10 (Techniques I and Techniques II) involve largely application skills that are intensive in mathematics and computations. Learning objective 9 (Techniques I) also assesses analytic skills. Lastly, learning objectives 6, 7, and 8 (Revenues I, Revenues II and Revenues III) contain a nice progression along the stages of cognitive reasoning. Revenues I contains questions that solely require knowledge, comprehension, and application. Revenues II assesses the latter three plus analytic skills. Revenues III is heavier on analytic skills. These distinctions are useful for the interpretation of results. Table 2. continued Specific learning objectives (continued) 5. Capital budgeting and debt finance (CapBud). The student should have a good understanding of (1) the equity implications of financing capital assets over their useful life, (2) the rationale for having separate capital budgets at the federal and state levels, (3) the different types of bonds and the activities that each is designed to finance, and (4) the basics of bond issuing and factors that raise or lower interest rate costs. 6. Revenues I: In general (RevI). The student should have a good understanding of basic concepts and computations for the diverse revenue sources for example, how to calculate taxes, how to determine effective tax rates, rules to determine business purchases that are exempt from sales taxation, etc. 7. Revenues II: Revenue evaluating criteria (RevII). The student should be able to evaluate individual revenue sources according to standard criteria (equity, efficiency, etc.) 8. Revenues III: Tradeoffs between major revenue sources (RevIII). The student should be able to evaluate the tradeoff between revenue sources (if one is substituted for the other) applying the same criteria. 9. Techniques I: Time value of money and its application in public finance (Tech1). The student should be competent in the application of quantitative techniques to account for time (i.e. productivity and inflation). Specifically, the student should be able to use these techniques to (1) create inflation indexes, (2) convert nominal to real expenditure and revenue trends, (3) project a current services budget, (4) determine rates of growth, (5) forecast revenues, (6) discount costs and benefits in cost benefit analysis, (7) determine the price of a bond, and (8) determine the assessed value of commercial property. 10. Techniques II: Basic forecasting techniques (TechII). The student should be competent in the application of the following basic revenue forecasting techniques: Linear trend (simple regression of revenues on time), moving averages, and proportionate rate of change. Journal of Public Affairs Education 419

8 Measuring learning objectives Having developed learning objectives the validity benchmark the next step is to develop the measuring instrument. A written multiple choice instrument, administered as midterm and final examinations, was used as the measuring instrument. The reasons for choosing this format and its limitations are discussed later. The first part of this step involved taking questions from previous exams and grouping them by learning objective. Two particularly important lessons were derived from this exercise. First, some objectives had a considerable amount of questions attached to them while other objectives had only one or two. In other words, some learning objectives were overrepresented while others were seriously underrepresented. This underscores and provides evidence for the validity issues raised above, which likely would have gone unnoticed had the process of defining and measuring learning objectives not been undertaken. Formally, one of the issues is known as content validity, which concerns the extent to which a measure adequately represents all facets of a concept (Singleton and Straits, 2005, 99). The other is mono-operation bias, which refers to using a single indicator to represent a construct: Construct validity will be lower in single exemplar research than in research where each construct is multiply operationalized in order to triangulate on the referent (Cook and Campbell, 1979, 65). These validity issues underscore the difficulty of designing valid exams in the absence of learning objectives. The second lesson was that several questions were unrelated to any relevant learning objective; this was likely the result of a goal-less exam writing process. For instance, questions are often taken from a text and are not related to a relevant learning objective, although they might represent useful information that can be applied in limited circumstances. To improve the validity of the measuring instrument, questions that were unrelated to learning objectives were eliminated and additional questions were written for underrepresented learning objectives. An effort was made to guarantee a minimum of five questions per learning objective, except for Techniques II, which was given only three questions corresponding to the three specific techniques covered in this section. The number of questions for each learning objective is given in Table 3. Basic guidelines for the design of multiple choice questions were followed; see Bradbard, Parker, and Stone (2004) for a summary. In addition, many of the questions had been tested on several occasions for their ability to discriminate between high and low scorers. Course-specific outcomes and benchmarking Once learning objectives are defined and measuring instruments are developed and administered, the third step of the method is to assess whether a particular benchmark has been achieved. The first part of this step involves processing results. Fortunately, most universities have Scantron processing centers that 420 Journal of Public Affairs Education

9 usually deliver results in one day, if not sooner. To save time, it is important to request output in an electronic file. With some minor data processing, results can be turned into a data set in which columns are learning objectives, rows are students, and each data entry is the percentage of correct answers that each student got for the questions under each learning objective. With the data arranged in this way, the competency of each student for each learning objective is measured as a percentage that can potentially range from 0 to 100 percent. A student that achieves a score of 100 percent for a particular learning objective can be considered fully competent in that particular set of knowledge, skills, and values. Results for the spring 2005 iteration of the public budgeting and finance course are presented in Table 3. This table presents descriptive statistics of the raw scores, with no curve or adjustments made. Final grades awarded to students may indeed be curved to judge each student against the average. However, it would be a mistake to make any kind of an adjustment to scores for purposes of gauging achievement of learning objectives. It is important for faculty and administrators to observe actual levels of competency achieved by students in the program under each learning objective. Simple descriptive statistics can provide valuable information about the assessment experience. Maximum scores, for instance, reveal that at least one student but not necessarily the same student is answering all questions correctly for each learning objective, except for the case of RevII where at least one student answered 90 percent of questions correctly. This is particularly good evidence Table 3. Descriptive Statistics and Number of Questions for Each Learning Objective Learning Objective Max Mean Median Min SD Number of Questions FPBF 100% 71% 75% 30% 20% 10 Bud 100% 83% 86% 50% 14% 14 Def 100% 70% 67% 17% 20% 6 PerfMeas 100% 73% 67% 44% 14% 9 CapBud 100% 59% 60% 20% 22% 5 RevI 100% 76% 77% 48% 15% 31 RevII 90% 68% 70% 30% 15% 10 RevIII 100% 65% 67% 17% 16% 6 TechI 100% 61% 55% 27% 21% 11 TechII 100% 35% 33% 0% 29% 3 N=28 students Journal of Public Affairs Education 421

10 that the exam is not excessively hard and that the top end of the distribution is fully meeting course and program objectives. In contrast, minimum scores show that at least some students are performing very poorly. Some of these minimum scores can be considered extreme outliers according to standard rules of thumb. Nevertheless, they are kept in the analysis, even if some of these students may have to repeat the course or may be unable to graduate on account of poor grades. The average score the measure of central location is the most informative about the level of achievement or competency of most students. It is also the average that is amenable to simple statistical inferential tools to judge whether a particular benchmark has been achieved. For this exercise, a benchmark or performance objective of 70 percent competency is chosen. This benchmark appears to pose a reasonable challenge, given open admission policies and the large spread in student academic backgrounds at FIU. To test whether this benchmark is reached for each learning objective, a simple one-sample t-test is conducted where H 0 : μ=70 percent. This procedure places the mean score for each learning objective within the repeated sampling framework of inferential statistics according to which the sample is only one of a theoretically infinite number of samples drawn from the same class. The purpose is simply to determine whether chance alone can explain the difference between the sample mean competency and the benchmark competency for the class (or hypothetical true mean). If chance is ruled out (i.e., p-values are smaller than the usual alpha levels) and the sample mean is less than 70 percent, the conclusion can be made that the benchmark has not been reached. The results of this analysis in the form of a report card are provided in Table 4. This report card, or a time series built from similar information collected over time, could be one way of presenting course grades as MPA program outcomes. Table 4 presents the mean competency, the t-score, the p-value, and a simple yes or no answer to whether the benchmark has been reached for each learning objective. If t-scores have negative signs and the corresponding p-values are smaller than the alpha level of.1, the conclusion is made that the benchmark has not been reached. For simplicity, a distinction is not made between the three alpha levels of.1,.05, and.01. For this class and this semester, the report card shows that three learning objectives have fallen below the benchmark: capital budgeting and debt finance (CapBud), time value of money techniques (TechI), and basic revenue forecasting techniques (TechII). Several relevant conclusions can be distilled from this report card, which may be valuable to individual faculty and the program. First, results show that average achievement or competency levels are not uniform across learning objectives. Second, results also show that students are on average doing more poorly in learning objectives that are challenging analytically and mathematically, specifically RevIII, TechI, and TechII. For individual faculty, results may point at areas 422 Journal of Public Affairs Education

11 where teaching effectiveness can be improved. For example, the information can be used to make decisions about how to allocate time and attention across learning objectives. This decision could entail giving more time and attention to failing learning objectives and, in exchange, taking time and attention from learning objectives where students clearly exceed the benchmark (Bud and RevI). However, time and attention alone may not be enough considering that failing learning objectives are more analytically and mathematically challenging. Changes in teaching methods, specifically active learning methods, may be required. Nevertheless, it must be said that a considerable amount of class time and active methods were dedicated to time value of money techniques (TechI) alone. This may suggest that changes in the program may also be necessary. From the standpoint of the program, results indicate areas in which the average student may be deficient as she or he enters the workforce. It may be particularly Table 4. Report Card for the Public Budgeting and Finance Course, Spring 2005 (Benchmark: Average competency of 70%) Learning Objective Average Competency t H 0 :μ=70% p-value Benchmark met? Foundations of public budgeting and finance (FPBF) 71% YES Federal and state budget processes (Bud) 83% YES Budget reform to control expenditures (Def) 70% YES Performance budgeting (PerfMeas) 73% YES Capital budgeting and debt finance (CapBud) 59% NO Revenues Conceptual and computational basics (RevI) 76% YES Revenues Evaluating criteria (RevII) 68% YES Revenue Tradeoffs between revenue sources (RevIII) 65% YES Techniques Time value of money (TechI) 61% NO Techniques Revenue forecasting techniques (TechII) 35% NO Note: Two-tail t-test. Benchmark is not met if t<0 and p<.1 Journal of Public Affairs Education 423

12 troubling that the average student may be deficient in time-value-of-money techniques, basic revenue forecasting, or critical thinking skills, given that these may be necessary for the average alumni to have access to the positions for which she or he is presumably being trained (city manager, finance director, and budget analyst). However, this evidence is from only one course of more than a dozen courses taken by students. Program administrators and the faculty as a whole will do well to take these results in the context of the entire program experience. Other courses such as quantitative methods, leadership, research methods, or the capstone experience may indeed be better suited for the development of critical thinking skills, or all courses taken together may produce a different overall outcome. In any event, results show that there is room for improvement. Several remedial actions could be considered at the program level. One of them is to establish an algebra refresher course for new students, which is used in other programs. A second measure is to establish certain prerequisites for the course. For example, taking a quantitative methods course prior to the public budgeting and finance class may help to improve outcomes. Lastly, a third measure is to cover certain essential skills in several courses throughout the curriculum to guarantee ample exposure and review by students. Limitations of the method This section discusses two limitations to the method presented here. The first derives from the use of multiple choice exams. The multiple choice format was considered the best alternative for two main reasons. First, writing and revising the exam can be time consuming; however, once written, it is quick and cheap to apply, score, and analyze using statistical tools. This is an important consideration if the systematic production of report cards is expected. Second, multiple choice exams are ideal for testing the broad range of knowledge, values, and skills taught in the introductory public budgeting and finance course. Indeed, considering the volume and diversity of knowledge, skill, and value sets covered in the public budgeting and finance class, it is hard to envision an alternative method that is as time efficient. One increasingly popular alternative for outcome assessment is the rubric approach (Stevens, 2005; Arter and McTigue, 2001). This method works well for courses with well-structured final papers or projects such as statistics, research methods, or the capstone course. However, given the extensive coverage of materials, it may not be the best option for the introductory course of public budgeting and finance Other frequently mentioned benefits of multiple choice exams are (1) reliability, (2) relative objectivity and low scoring bias, (3) low measurement error, (4) high correlation with scores of constructed-response (short essay) exams, and (5) when well-designed, adequacy to test at least some higher-order or critical thinking skills (Chan and Kennedy, 2002; Buckles and Siegfried, 2006; Dogan Tozoglu and Ahmet Gurses, 2004; Bradbard, Parker, and Stone, 2004). However, 424 Journal of Public Affairs Education

13 tradeoffs arise from the use of the multiple-choice format, particularly in comparison to the so-called constructed-response (short answer/essay) format. Disadvantages of multiple choice exams include facilitating guessing, testing partial knowledge, and lower degree of difficulty (Chan and Kennedy, 2002; Bradbard, Parker, and Stone, 2004). Although some of these problems can be attenuated by improving the design of questions, including as many response options as possible, or using alternate ways of scoring (Reid, 1976; Bradbard, Parker, and Stone, 2004), some scholars simply prefer constructed-response formats because they are believed to measure more important skills, be more relevant to applied decision making, better reflect changing social values, and have more positive social consequences (Katz, Bennett, and Berger, 2000, 55). The second limitation of the method is more fundamental. Even though the multiple-choice format allows for the inclusion of large samples of course content, measured outcomes do not include the entire classroom experience. The spring 2005 public budgeting and finance course in FIU s public administration program, in addition to midterm and final exams, included regular assignments, a guest speaker, class discussions, and a final project where students formed teams, chose a city or other local entity, interviewed public officials, analyzed budget and financial documents, and finally performed an investigation and assessment of expenditure, revenue, and financial management processes, practices, trends, and challenges. The learning experience in this final project has been well-rated by students and has even led to job offers. However, much of what students may have learned in these additional deliverables and activities is not captured in the report card produced in this article. The omission may be particularly unfortunate if one considers that the educational objectives of these deliverables and activities may be rich in critical thinking skills. This discussion underscores the difficulty of producing perfect program outcomes, a constant complaint in the field. Although solace can be taken in David G. Williams s suggestion to not let the perfect drive out the good (2002, 45), these limitations could motivate the development of complementary indicators to capture the missing part of the classroom experience. Conclusion MPA programs have been under increasing pressure from external stakeholders to produce indicators of program effectiveness. Producing these measures, however, can be both difficult and expensive (Williams, 2002; Jennings, 1989). This article presents a low-cost method of transforming grades into meaningful indicators of program effectiveness, with ancillary benefits in the areas of faculty and program development. The simple method presented in this paper may, in addition to improving teaching effectiveness, provide timely information to administrators and faculty about the level of student competency in knowledge, skills, and values considered essential for a successful career in public service. Journal of Public Affairs Education 425

The University of Texas at San Antonio 1 Department of Public Administration The Department of Public Administration offers the Master of Public Administration, as well as the Graduate Certificate in Nonprofit

MASTER OF PUBLIC ADMINISTRATION Northern Illinois University The Master of Public Administration (MPA) degree program at Northern Illinois University offers specialized educational preparation for students

TEAC Quality Principles for Educational Leadership Faculty members seeking TEAC accreditation for their programs in educational leadership must affirm that their goal is to prepare competent, caring, and

Public Administration (M.P.A.) Public Administration Admission The M.P.A. degree is the recognized professional degree in public administration and is appropriate for students pursuing public service careers

Three Principles of Competency-Based Learning: Mission, Mission, Mission William C. Rivenbark and Willow S. Jacobson University of North Carolina at Chapel Hill ABSTRACT NASPAA adopted new accreditation

Manual for the Master of Public Administration Degree College for Design and Social Inquiry Florida Atlantic University www.fau.edu/spa/mpa Make a difference through public service. THE MASTER OF PUBLIC

602 STRATEGIC PLANNING AND ORGANIZATIONAL CHANGE IN THE PUBLIC AND NONPROFIT SECTORS. (3) This course focuses on the potential for change and future directions for public and nonprofit organizations. It

WASHBURN UNIVERSITY DUAL DEGREE PROGRAM School of Law (J.D.) & School of Business (M.B.A.) Recognizing that legal and business studies complement each other and that real-world problems often require solutions

Master of Public Administration: a NASPAA Accredited Program 2010 2011 NASPAA Self Study Report The National Association of Schools of Public Affairs and Administration (NASPAA) requires member programs

Master of Arts concentration in Human Resource Management Coordinator: Advisor: Dr. Michael Harrison Dr. Robert Awkward The Master of Arts with a concentration in is designed to provide qualified individuals

MPA Program Assessment Report Summer 2015 Introduction: This was the second full year for doing learning outcomes assessment based on the 2009 NASPAA accreditation standards and conducting our exit interviews

Higher Education Dickinson Hall, Room 419 (501) 569-3267 Master of Arts and Doctor of Education The Master of Arts in Higher Education is designed for those individuals who are interested in entering or

Guidance for PA/H 509 Field Supervisors College of Urban and Public Affairs Mark O. Hatfield School of Government Division of Public Administration P.O. Box 751 Portland, OR 97207-0751 506 SW Mill St.,

f ce.in the SchooLof LiberalArt Public Administration, B.A. Effectively assessing a degree program should address a number of factors: ---------"Lj-Valid_studenLieaming cutcomes shouidbe clearl'i articulated;

NORTHERN ILLINOIS UNIVERSITY College: College of Business Department: Inter-Departmental Program: Master of Business Administration CIP Code: 52.0201 Northern Illinois University s M.B.A. program follows

School of Advanced Studies Doctor Of Management In Organizational Leadership/information Systems And Technology The mission of the Information Systems and Technology specialization of the Doctor of Management

Department of Accounting, Finance, & Economics Assessment Report 2010-2011 Mission/Purpose The mission of the Department of Accounting, Finance and Economics is to provide a quality education in accounting,

The Case for EMBA Sponsorship Part 1 - Preparation and Research Written by Daniel Szpiro, Dean of Executive Education at The Jack Welch Management Institute http://community.mbaworld.com/blog/b/weblog/archive/2015/07/31/thecase-for-emba-sponsorship

Areas of Concentration: Master of Public Administration Academic Year 2015-2016 Department of Public Administration and Policy School of Public Affairs American University Mission Statement The mission

FROM GRADUATE CATALOG 2013-2015 COLLEGE OF PUBLIC SERVICE AND URBAN AFFAIRS Michael Harris, Ph.D., Dean Office: Suite, F-400, Avon William Campus (615) 963-7201 mharris50@tnstate.edu DEPARTMENT OF PUBLIC

Assessing Quantitative Reasoning in GE (Owens, Ladwig, and Mills) Introduction Many students at CSU, Chico, receive much of their college-level mathematics education from the one MATH course they complete

Master of Public Administration Graduate Program The Barbara Jordan Mickey Leland School of Public Affairs Our Program O u r Pr o g r a m The Barbara Jordan Mickey Leland School of Public Affairs Program

Program Overview This accelerated program provides candidates with an opportunity to develop the critical leadership skills and knowledge that are required in today s increasingly complex, diverse, and

STATEMENT ON Counselor Competencies Approved by the Executive Board, July 2000 Introduction The National Association for College Admission Counseling (NACAC) believes there is a basic body of knowledge

Master of Health Care Administration Program Strategic Plan 2008 2010 The faculty and staff of the MHA program met and developed the following document. Included are foundational statements that include:

Program Overview The PhD in Business Leadership and Administration is designed for professionals located nation wide who desire an advanced degree in business to excel in their careers. In addition, the

284 MASTER OF BUSINESS ADMINISTRATION PROGRAM The Clarke University MBA Program seeks to attract professionals that are interested in ethical business practices and lifelong learning. In addition to the

School of Advanced Studies Doctor Of Business Administration The mission of the Doctor of Business Administration degree program is to empower business administration practitioners to become business subject

Master of Business Administration Information Technology Management Emphasis The Master of Business Administration Information Technology Management is specifically designed for experienced business professionals

THE MASTER OF ARTS PROGRAM IN INDUSTRIAL/ORGANIZATIONAL PSYCHOLOGY GRADUATE SCHOOL OF ARTS AND SCIENCE NEW YORK UNIVERSITY Information Booklet for Applicants Director: Barry H. Cohen, Ph.D Telephone: 212-998-7815

Program Report for the Preparation of Elementary School Teachers Association for Childhood Education International (ACEI) 2007 Standards - Option A NATIONAL COUNCIL FOR ACCREDITATION OF TEACHER EDUCATION

Academic Program Assessment Plan Certificate: Homeland Security Studies Department of History and Political Science College of Humanities and Social Sciences Colorado State University-Pueblo Plan appended

2005 ABET Program Outcome and Assessment 3. Program Outcomes and Assessment Program Outcomes We define our program educational outcomes as statements that describe what students are expected to be able

University of Northern Iowa College of Business Administration Master of Business Administration Learning Assurance Program Last updated April 2009 1. Philosophy and Program Goals The MBA program s Learning

UNF-MPA student learning outcomes and program assessment Prepared by Dr. G.G. Candler MPA Director 7 May 2014 UNF MPA program assessment includes a number of elements: 1. An alumni survey, 2. a student

Clayton State University Master of Health Administration Program Fall 2009 Assessment MHA Program Mission: The mission of the Health Care Management program is to prepare ethical and socially responsible

ENVIRONMENTAL HEALTH SCIENCE ASSESSMENT REPORT 2009-2010 The Environmental Health Science (EHS) Program at East Central University is one of 30 accredited undergraduate environmental health programs in

Dubai School of Government Master of Public Administration DSG MPA Introduction The Dubai School of Government is a research and teaching institution focusing on public policy in the Arab world. Established

BS Environmental Science (2013-2014) Program Information Point of Contact Brian M. Morgan (brian.morgan@marshall.edu) Support for University and College Missions Marshall University is a multi-campus public

As part of our ACBSP accreditation requirements, we routinely provide reliable information to the public on our performance, including student achievement such as assessment results. Portions of our report

AACSB Standards: Assessment of a Finance Program Nell S. Gullett, The University of Tennessee at Martin Arnold L. Redman, The University of Tennessee at Martin ABSTRACT The Association to Advance Collegiate

Curriculum - Doctor of Philosophy CORE COURSES Pharm 545-546.Pharmacoeconomics, Healthcare Systems Review. (3, 3) Exploration of the cultural foundations of pharmacy. Development of the present state of

DOCTORAL PROGRAM IN HIGHER EDUCATION Community College Cognate University of North Texas Contact: Patsy Fulton-Calkins, Ph.D. Director of the Bill J. Priest Center for Community College Education Don A.

School of Advanced Studies Doctor Of Health Administration The mission of the Doctor of Health Administration degree program is to develop healthcare leaders by educating them in the areas of active inquiry,

OKLAHOMA STATE UNIVERSITY School of Applied Health and Educational Psychology Ph.D. Program in Educational Psychology Program Description for Educational Psychology Option School Mission The mission of

University of the District of Columbia School of Engineering and Applied Sciences Continuous Improvement Process And ABET Outcomes Assessment Protocol BASED ON ABET 2008-2009 Criteria April 2008 School