Academic Commons Search Resultshttp://academiccommons.columbia.edu/catalog.rss?f%5Bauthor_facet%5D%5B%5D=Kaiser%2C+Gail+E.&f%5Blanguage%5D%5B%5D=English&q=&rows=500&sort=record_creation_date+desc
Academic Commons Search Resultsen-usPhosphor: Illuminating Dynamic Data Flow in the JVM (Artifact for Evaluation)http://academiccommons.columbia.edu/catalog/ac:182689
Bell, Jonathan Schaffer; Kaiser, Gail E.http://dx.doi.org/10.7916/D84F1PH4Tue, 13 Jan 2015 00:00:00 +0000Dynamic taint analysis is a well-known information flow analysis problem with many possible applications. Taint tracking allows for analysis of application data flow by assigning labels to inputs, and then propagating those labels through data flow. Taint tracking systems traditionally compromise among performance, precision, accuracy, and portability. Performance can be critical, as these systems are typically intended to be deployed with software, and hence must have low overhead. To be deployed in security-conscious settings, taint tracking must also be accurate and precise. Dynamic taint tracking must be portable in order to be easily deployed and adopted for real world purposes, without requiring recompilation of the operating system or language interpreter, and without requiring access to application source code. We present Phosphor, a dynamic taint tracking system for the Java Virtual Machine (JVM) that simultaneously achieves our goals of performance, accuracy, precision, and portability. Moreover, to our knowledge, it is the first portable general purpose taint tracking system for the JVM. We evaluated Phosphor's performance on two commonly used JVM languages (Java and Scala), on two versions of two commonly used JVMs (Oracle's HotSpot and OpenJDK's IcedTea) and on Android's Dalvik Virtual Machine, finding its performance to be impressive: as low as 3% (53% on average), using the DaCapo macro benchmark suite. This artifact contains the code needed to reproduce the experiments detailed in our paper.Computer sciencejsb2125, gek1Computer ScienceComputer softwareFailure Analysis of the New York City Power Gridhttp://academiccommons.columbia.edu/catalog/ac:179086
Wu, Leon; Anderson, Roger N.; Boulanger, Albert G.; Rudin, Cynthia; Kaiser, Gail E.http://dx.doi.org/10.7916/D8G73CBZWed, 29 Oct 2014 00:00:00 +0000As U.S. power grid transforms itself into Smart Grid, it has become less reliable in the past years. Power grid failures lead to huge financial cost and affect people’s life. Using a statistical analysis and holistic approach, this paper analyzes the New York City power grid failures: failure patterns and climatic effects. Our findings include: higher peak electrical load increases likelihood of power grid failure; increased subsequent failures among electrical feeders sharing the same substation; underground feeders fail less than overhead feeders; cables and joints installed during certain years are more likely to fail; higher weather temperature leads to more power grid failures. We further suggest preventive maintenance, intertemporal consumption, and electrical load optimization for failure prevention. We also estimated that the predictability of the power grid component failures correlates with the cycles of the North Atlantic Oscillation (NAO) Index.Computer sciencellw2107, rna1, agb6, gek1Computer Science, Center for Computational Learning SystemsTechnical reportsPhosphor: Illuminating Dynamic Data Flow in the JVMhttp://academiccommons.columbia.edu/catalog/ac:174980
Bell, Jonathan Schaffer; Kaiser, Gail E.http://dx.doi.org/10.7916/D8QJ7FFXTue, 17 Jun 2014 00:00:00 +0000Dynamic taint analysis is a well-known information flow analysis problem with many possible applications. Taint tracking allows for analysis of application data flow by assigning labels to inputs, and then propagating those labels through data flow. Taint tracking systems traditionally compromise among performance, precision, accuracy, and portability. Performance can be critical, as these systems are typically intended to be deployed with software, and hence must have low overhead. To be deployed in security-conscious settings, taint tracking must also be accurate and precise. Dynamic taint tracking must be portable in order to be easily deployed and adopted for real world purposes, without requiring recompilation of the operating system or language interpreter, and without requiring access to application source code. We present Phosphor, a dynamic taint tracking system for the Java Virtual Machine (JVM) that simultaneously achieves our goals of performance, accuracy, precision, and portability. Moreover, to our knowledge, it is the first portable general purpose taint tracking system for the JVM. We evaluated Phosphor's performance on two commonly used JVM languages (Java and Scala), on two versions of two commonly used JVMs (Oracle's HotSpot and OpenJDK's IcedTea) and on Android's Dalvik Virtual Machine, finding its performance to be impressive: as low as 3% (53% on average), using the DaCapo macro benchmark suite. This paper describes the approach that Phosphor uses to achieve portable taint tracking in the JVM.Computer sciencejsb2125, gek1Computer ScienceTechnical reportsN Heads Are Better Than Nonehttp://academiccommons.columbia.edu/catalog/ac:167858
Hopkins, Morris; Casteneda, Mauricio; Sheth, Swapneel Kalpesh; Kaiser, Gail E.http://dx.doi.org/10.7916/D8028PFVWed, 27 Nov 2013 00:00:00 +0000Social network platforms have transformed how people communicate and share information. However, as these platforms have evolved, the ability for users to control how and with whom information is being shared introduces challenges concerning the configuration and comprehension of privacy settings. To address these concerns, our crowd sourced approach simplifies the understanding of privacy settings by using data collected from 512 users over a 17 month period to generate visualizations that allow users to compare their personal settings to an arbitrary subset of individuals of their choosing. To validate our approach we conducted an online survey with closed and open questions and collected 59 valid responses after which we conducted follow-up interviews with 10 respondents. Our results showed that 70% of respondents found visualizations using crowd sourced data useful for understanding privacy settings, and 80% preferred a crowd sourced tool for configuring their privacy settings over current privacy controls.Computer sciencemah2250, mc3683, sks2142, gek1Computer ScienceTechnical reportsUs and Them - A Study of Privacy Requirements Across North America, Asia, and Europehttp://academiccommons.columbia.edu/catalog/ac:167855
Sheth, Swapneel Kalpesh ; Kaiser, Gail E.http://dx.doi.org/10.7916/D8028PFVWed, 27 Nov 2013 00:00:00 +0000Data privacy when using online systems like Facebook and Amazon has become an increasingly popular topic in the last few years. However, only a little is known about how users and developers perceive privacy and which concrete measures would mitigate privacy concerns. To investigate privacy requirements, we conducted an online survey with closed and open questions and collected 408 valid responses. Our results show that users often reduce privacy to security, with data sharing and data breaches being their biggest concerns. Users are more concerned about the content of their documents and personal data such as location than their interaction data. Unlike users, developers clearly prefer technical measures like data anonymization and think that privacy laws and policies are less effective. We also observed interesting differences between people from different geographies. For example, people from Europe are more concerned about data breaches than people from North America. People from Asia/Pacific and Europe believe that content and metadata are more critical for privacy than people from North America. Our results contribute to developing a user-driven privacy framework that is based on empirical evidence in addition to the legal, technical, and commercial perspectives.Computer sciencesks2142, gek1Computer ScienceTechnical reportsUnit Test Virtualization with VMVMhttp://academiccommons.columbia.edu/catalog/ac:165654
Bell, Jonathan Schaffer; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:21764Mon, 23 Sep 2013 00:00:00 +0000Testing large software packages can become very time intensive. To address this problem, researchers have investigated techniques such as Test Suite Minimization. Test Suite Minimization reduces the number of tests in a suite by removing tests that appear redundant, at the risk of reducing fault-finding ability since it can be difficult to identify which tests are truly redundant. We take a completely different approach to solving the same problem of long running test suites by instead reducing the time needed to execute each test, an approach that we call Unit Test Virtualization. We describe the empirical analysis that we performed to ground our approach and provide an implementation of Unit Test Virtualization targeting Java applications. We evaluated our implementation, VMVM, using 20 real-world Java applications and found that it reduces test suite execution time by up to 97 percent (on average, 62 percent) when compared to traditional unit test execution. We also compared VMVM to a well known Test Suite Minimization technique, finding the reduction provided by VMVM to be four times greater, while still executing every test with no loss of fault-finding ability.Computer sciencejsb2125, gek1Computer ScienceTechnical reportsEffectiveness of Teaching Metamorphic Testing, Part IIhttp://academiccommons.columbia.edu/catalog/ac:165332
Mishra, Kunal S.; Kaiser, Gail E.; Sheth, Swapneel K.http://hdl.handle.net/10022/AC:P:21693Thu, 19 Sep 2013 00:00:00 +0000We study the ability of students in a senior/graduate software engineering course to understand andapply metamorphic testing, a relatively recently invented advance in software testing research that complements conventional approaches such as equivalence partitioning and boundary analysis. We previously reported our investigation of the fall 2011 offering of the Columbia University course COMS W4156 Advanced Software Engineering, and here report on the fall 2012 offering and contrast it to the previous year. Our main findings are: 1) Although the students in the second offering did not do very well on the newly added individual assignment specifically focused on metamorphic testing, thereafter they were better able to find metamorphic properties for their team projects than the students from the previous year who did not have that preliminary homework and, perhaps most significantly, did not have the solution set for that homework. 2) Students in the second offering did reasonably well using the relatively novel metamorphic testing technique vs. traditional black box testing techniques in their projects (such comparison data is not available for the first offering). 3) Finally, in both semesters, the majority of the student teams were able to apply metamorphic testing to their team projects after only minimal instruction, which would imply that metamorphic testing is a viable strategy for student testers.Computer scienceksm2135, gek1, sks2142Computer ScienceTechnical reportsMetamorphic Runtime Checking of Applications without Test Oracleshttp://academiccommons.columbia.edu/catalog/ac:165342
Murphy, Christian; Kaiser, Gail E.; Bell, Jonathan Schaffer; Su, Fang-hsianghttp://hdl.handle.net/10022/AC:P:21695Thu, 19 Sep 2013 00:00:00 +0000Challenges arise in testing applications that do not have test oracles, i.e., for which it is impossible or impractical to know what the correct output should be for general input. Metamorphic testing, introduced by Chen et al., has been shown to be a simple yet effective technique in testing these types of applications: test inputs are transformed in such a way that it is possible to predict the expected change to the output, and if the output resulting from this transformation is not as expected, then a fault must exist. Here, we improve upon previous work by presenting a new technique called Metamorphic Runtime Checking, which automatically conducts metamorphic testing of both the entire application and individual functions during a program's execution. This new approach improves the scope, scale, and sensitivity of metamorphic testing by allowing for the identification of more properties and execution of more tests, and increasing the likelihood of detecting faults not be found by application-level properties alone. We also discuss a technique for automatically discovering functions' metamorphic properties, and present the results of new studies that demonstrate that Metamorphic Runtime Checking advances the state of the art in testing applications without oracles.Computer sciencegek1, jsb2125, fs2455Computer ScienceTechnical reportsAn Architecture for Multi-User Software Development Environmentshttp://academiccommons.columbia.edu/catalog/ac:164119
Ben-Shaul, Israel Z.; Kaiser, Gail E.; Heineman, George E.http://hdl.handle.net/10022/AC:P:21281Wed, 07 Aug 2013 00:00:00 +0000We present an architecture for multi-user software development environments, covering general, process-centered and rule-based MUSDEs. Our architecture is founded on componentization, with particular concern for the capability to replace the synchronization component - to allow experimentation with novel concurrency control mechanisms - with minimal effects on other components while still supporting integration. The architecture has been implemented in the MARVEL SDEComputer sciencegek1Computer ScienceTechnical reportsChroniclerJ: Lightweight Recording of Nondeterministic Inputs to Reproduce Field Failures (System Implementation)http://academiccommons.columbia.edu/catalog/ac:162688
Bell, Jonathan Schaffer; Sarda, Nikhil; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:20847Thu, 27 Jun 2013 00:00:00 +0000This archive contains the Java implementation of our system, Chronicler, described below: When programs fail in the field, developers are often left with limited information to diagnose the failure. Automated error reporting tools can assist in bug report generation but without precise steps from the end user it is often difficult for developers to recreate the failure. Advanced remote debugging tools aim to capture sufficient information from field executions to recreate failures in the lab but often have too much overhead to practically deploy. We present CHRONICLER, an approach to remote debugging that captures non-deterministic inputs to applications in a lightweight manner, assuring faithful reproduction of client executions. We evaluated CHRONICLER by creating a Java implementation, CHRONICLERJ, and then by using a set of benchmarks mimicking real world applications and workloads, showing its runtime overhead to be under 10% in most cases (worst case 8! 6%), while an existing tool showed overhead over 100% in the same cases (worst case 2,322%).Information technology, Computer sciencejsb2125, ns2847, gek1Computer ScienceComputer softwareFARE: A Framework for Benchmarking Reliability of Cyber-Physical Systemshttp://academiccommons.columbia.edu/catalog/ac:158876
Wu, Leon; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:19679Fri, 05 Apr 2013 00:00:00 +0000A cyber-physical system (CPS) is a system featuring a tight combination of, and coordination between, the system's computational and physical elements. System reliability is a critical requirement of cyber-physical systems. An unreliable CPS often leads to system malfunctions, service disruptions, financial losses and even human life. Improving CPS reliability requires an objective measurement, estimation and comparison of the CPS system reliability. This paper describes FARE (Failure Analysis and Reliability Estimation), a framework for benchmarking reliability of cyber-physical systems. Some prior researches have proposed reliability benchmark for some specific CPS such as wind power plant and wireless sensor networks. There were also some prior researches on the components of CPS such as software and some specific hardware. But according to the best of our knowledge, there isn't any reliability benchmark framework for CPS in general. FARE framework provides a CPS reliability model, a set of methods and metrics on the evaluation environment selection, failure analysis and reliability estimation for benchmarking CPS reliability. It not only provides a retrospect evaluation and estimation of the CPS system reliability using the past data, but also provides a mechanism for continuous monitoring and evaluation of CPS reliability for runtime enhancement. The framework is extensible for accommodating new reliability measurement techniques and metrics. It is also generic and applicable to a wide range of CPS applications. For empirical study, we applied the FARE framework on a smart building management system for a large commercial building in New York City. Our experiments showed that FARE is easy to implement, accurate for comparison and can be used for building useful industry benchmarks and standards after accumulating enough data.Computer sciencellw2107, gek1Computer ScienceTechnical reportsIncreasing Student Engagement in Software Engineering with Gamificationhttp://academiccommons.columbia.edu/catalog/ac:154509
Sheth, Swapneel Kalpesh; Bell, Jonathan Schaffer; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:15273Tue, 13 Nov 2012 00:00:00 +0000Gamification, or the use of game elements in non-game contexts, has become an increasingly popular approach to increasing end-user engagement in many contexts, including employee productivity, sales, recycling, and education. Our preliminary work has shown that gamification can be used to boost student engagement and learning in basic software testing. We seek to expand our gamified software engineering approach to motivate other software engineering best practices. We propose to build a game layer on top of traditional continuous integration technologies to increase student engagement in development, documentation, bug reporting, and test coverage. This poster describes to our approach and presents some early results showing feasability.Computer sciencesks2142, jsb2125, gek1Computer ScienceTechnical reportsA Competitive-Collaborative Approach for Introducing Software Engineering in a CS2 Classhttp://academiccommons.columbia.edu/catalog/ac:154506
Sheth, Swapneel Kalpesh; Bell, Jonathan Schaffer; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:15272Tue, 13 Nov 2012 00:00:00 +0000Introductory Computer Science (CS) classes are typically competitive in nature. The cutthroat nature of these classes comes from students attempting to get as high a grade as possible, which may or may not correlate with actual learning. Further, there is very little collaboration allowed in most introductory CS classes. Most assignments are completed individually since many educators feel that students learn the most, especially in introductory classes, by working alone. In addition to completing "normal" individual assignments, which have many benefits, we wanted to expose students to collaboration early (via, for example, team projects). In this paper, we describe how we leveraged competition and collaboration in a CS2 to help students learn aspects of computer science better — in this case, good software design and software testing — and summarize student feedback.Computer sciencesks2142, jsb2125, gek1Computer ScienceTechnical reportsEnd-User Regression Testing for Privacyhttp://academiccommons.columbia.edu/catalog/ac:152841
Sheth, Swapneel Kalpesh; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:14776Wed, 26 Sep 2012 00:00:00 +0000Privacy in social computing systems has become a major concern. End-users of such systems find it increasingly hard to understand complex privacy settings. As software evolves over time, this might introduce bugs that breach users' privacy. Further, there might be system-wide policy changes that could change users' settings to be more or less private than before. We present a novel technique that can be used by end-users for detecting changes in privacy, i.e., regression testing for privacy. Using a social approach for detecting privacy bugs, we present two prototype tools. Our evaluation shows the feasibility and utility of our approach for detecting privacy bugs. We highlight two interesting case studies on the bugs that were discovered using our tools. To the best of our knowledge, this is the first technique that leverages regression testing for detecting privacy bugs from an end-user perspective.Computer sciencesks2142, gek1Computer Science, Electrical EngineeringTechnical reportsAn Autonomic Reliability Improvement System for Cyber-Physical Systemshttp://academiccommons.columbia.edu/catalog/ac:152586
Wu, Leon Li; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:14722Thu, 20 Sep 2012 00:00:00 +0000System reliability is a fundamental requirement of cyber-physical systems. Unreliable systems can lead to disruption of service, financial cost and even loss of human life. Typical cyber-physical systems are designed to process large amounts of data, employ software as a system component, run online continuously and retain an operator-in-the-loop because of human judgment and accountability requirements for safety-critical systems. This paper describes a data-centric runtime monitoring system named ARIS (Autonomic Reliability Improvement System) for improving the reliability of these types of cyber-physical systems. ARIS employs automated online evaluation, working in parallel with the cyber-physical system to continuously conduct automated evaluation at multiple stages in the system workflow and provide real-time feedback for reliability improvement. This approach enables effective evaluation of data from cyber-physical systems. For example, abnormal input and output data can be detected and flagged through data quality analysis. As a result, alerts can be sent to the operator-in-the-loop, who can then take actions and make changes to the system based on these alerts in order to achieve minimal system downtime and higher system reliability. We have implemented ARIS in a large commercial building cyber-physical system in New York City, and our experiment has shown that it is effective and efficient in improving building system reliability.Computer sciencellw2107, gek1Computer Science, Center for Computational Learning SystemsTechnical reportsChronicler: Lightweight Recording to Reproduce Field Failureshttp://academiccommons.columbia.edu/catalog/ac:151789
Bell, Jonathan Schaffer; Sarda, Nikhil; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:14436Fri, 17 Aug 2012 00:00:00 +0000When programs fail in the field, developers are often left with limited information to diagnose the failure. Automated error reporting tools can assist in bug report generation but without precise steps from the end user it is often difficult for developers to recreate the failure. Advanced remote debugging tools aim to capture sufficient information from field executions to recreate failures in the lab, but have too much overhead to practically deploy. We present CHRONICLER, an approach to remote debugging that captures nondeterministic inputs to applications in a lightweight manner, guaranteeing faithful reproduction of client executions. We evaluated CHRONICLER by creating a Java implementation, CHRONICLERJ, and then by using a set of benchmarks mimicking real world applications and workloads, showing its runtime overhead to be under 10% in most cases (worst case 86%), while an existing tool showed overhead over 100% in the same cases (worst case 2,322%).Computer sciencejsb2125, ns2847, gek1Computer ScienceTechnical reportsA Holistic Approach to Service Survivabilityhttp://academiccommons.columbia.edu/catalog/ac:149556
Keromytis, Angelos D.; Parekh, Janak J.; Gross, Philip N.; Kaiser, Gail E.; Misra, Vishal; Nieh, Jason; Rubenstein, Daniel Stuart; Stolfo, Salvatorehttp://hdl.handle.net/10022/AC:P:13979Wed, 11 Jul 2012 00:00:00 +0000We present SABER (Survivability Architecture: Block, Evade, React), a proposed survivability architecture that blocks, evades and reacts to a variety of attacks by using several security and survivability mechanisms in an automated and coordinated fashion. Contrary to the ad hoc manner in which contemporary survivable systems are built-using isolated, independent security mechanisms such as firewalls, intrusion detection systems and software sandboxes-SABER integrates several different technologies in an attempt to provide a unified framework for responding to the wide range of attacks malicious insiders and outsiders can launch. This coordinated multi-layer approach will be capable of defending against attacks targeted at various levels of the network stack, such as congestion-based DoS attacks, software-based DoS or code-injection attacks, and others. Our fundamental insight is that while multiple lines of defense are useful, most conventional, uncoordinated approaches fail to exploit the full range of available responses to incidents. By coordinating the response, the ability to survive successful security breaches increases substantially. We discuss the key components of SABER, how they will be integrated together, and how we can leverage on the promising results of the individual components to improve survivability in a variety of coordinated attack scenarios. SABER is currently in the prototyping stages, with several interesting open research topics.Computer scienceak2052, png3, gek1, vm2020, jn234, dsr100, sjs11Computer ScienceArticlesImproving Efficiency and Reliability of Building Systems Using Machine Learning and Automated Online Evaluationhttp://academiccommons.columbia.edu/catalog/ac:146877
Wu, Leon Li; Kaiser, Gail E.; Solomon, David M.; Winter, Rebecca Lynn; Boulanger, Albert G.; Anderson, Roger N.http://hdl.handle.net/10022/AC:P:13213Thu, 10 May 2012 00:00:00 +0000A high percentage of newly-constructed commercial office buildings experience energy consumption that exceeds specifications and system failures after being put into use. This problem is even worse for older buildings. We present a new approach, 'predictive building energy optimization', which uses machine learning (ML) and automated online evaluation of historical and real-time building data to improve efficiency and reliability of building operations without requiring large amounts of additional capital investment. Our ML approach uses a predictive model to generate accurate energy demand forecasts and automated analyses that can guide optimization of building operations. In parallel, an automated online evaluation system monitors efficiency at multiple stages in the system workflow and provides building operators with continuous feedback. We implemented a prototype of this application in a large commercial building in Manhattan. Our predictive machine learning model applies Support Vector Regression (SVR) to the building's historical energy use and temperature and wet-bulb humidity data from the building's interior and exterior in order to model performance for each day. This predictive model closely approximates actual energy usage values, with some seasonal and occupant-specific variability, and the dependence of the data on day-of-the-week makes the model easily applicable to different types of buildings with minimal adjustment. In parallel, an automated online evaluator monitors the building's internal and external conditions, control actions and the results of those actions. Intelligent real-time data quality analysis components quickly detect anomalies and automatically transmit feedback to building management, who can then take necessary preventive or corrective actions. Our experiments show that this evaluator is responsive and effective in further ensuring reliable and energyefficient operation of building systems.Computer sciencellw2107, gek1, rlw2152, agb6, rna1Computer Science, Earth and Environmental Engineering, Center for Computational Learning Systems, Earth and Environmental SciencesTechnical reportsConstructing Subtle Concurrency Bugs Using Synchronization-Centric Second-Order Mutation Operatorshttp://academiccommons.columbia.edu/catalog/ac:146615
Wu, Leon Li; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:13131Thu, 03 May 2012 00:00:00 +0000Mutation testing applies mutation operators to modify program source code or byte code in small ways, and then runs these modified programs (i.e., mutants) against a test suite in order to evaluate the quality of the test suite. In this paper, we first describe a general fault model for con- current programs and some limitations of previously developed sets of first-order concurrency mutation operators. We then present our new mutation testing approach, which employs synchronization-centric second-order mutation operators that are able to generate subtle concurrency bugs not represented by the first-order mutation. These operators are used in addition to the synchronization-centric first-order mutation operators to form a small set of effective concurrency mutation operators for mutant generation. Our empirical study shows that our set of operators is effective in mutant generation with limited cost and demonstrates that this new approach is easy to implement.Computer sciencellw2107, gek1Computer Science, Center for Computational Learning SystemsTechnical reportsBUGMINER: Software Reliability Analysis Via Data Mining of Bug Reportshttp://academiccommons.columbia.edu/catalog/ac:146618
Wu, Leon Li; Xie, Boyi; Kaiser, Gail E.; Passonneau, Rebeccahttp://hdl.handle.net/10022/AC:P:13132Thu, 03 May 2012 00:00:00 +0000Software bugs reported by human users and automatic error reporting software are often stored in some bug tracking tools (e.g., Bugzilla and Debbugs). These accumulated bug reports may contain valuable information that could be used to improve the quality of the bug reporting, reduce the quality assurance effort and cost, analyze software reliability, and predict future bug report trend. In this paper, we present BUGMINER, a tool that is able to derive useful information from historic bug report database using data mining, use these information to do completion check and redundancy check on a new or given bug report, and to estimate the bug report trend using statistical analysis. Our empirical studies of the tool using several real-world bug report repositories show that it is effective, easy to implement, and has relatively high accuracy despite low quality data.Computer sciencellw2107, bx2109, gek1, rp34Computer Science, Center for Computational Learning SystemsTechnical reportsEvaluating Machine Learning for Improving Power Grid Reliabilityhttp://academiccommons.columbia.edu/catalog/ac:146622
Wu, Leon Li; Kaiser, Gail E.; Rudin, Cynthia; Waltz, David L.; Anderson, Roger N.; Boulanger, Albert G.; Salleb-Aouissi, Ansaf; Dutta, Haimonti; Pooleery, Manojhttp://hdl.handle.net/10022/AC:P:13133Thu, 03 May 2012 00:00:00 +0000Ensuring reliability as the electrical grid morphs into the "smart grid" will require innovations in how we assess the state of the grid, for the purpose of proactive maintenance, rather than reactive maintenance; in the future, we will not only react to failures, but also try to anticipate and avoid them using predictive modeling (machine learning and data mining) techniques. To help in meeting this challenge, we present the Neutral Online Visualization-aided Autonomic evaluation framework (NOVA) for evaluating machine learning and data mining algorithms for preventive maintenance on the electrical grid. NOVA has three stages provided through a unified user interface: evaluation of input data quality, evaluation of machine learning and data mining results, and evaluation of the reliability improvement of the power grid. A prototype version of NOVA has been deployed for the power grid in New York City, and it is able to evaluate machine learning and data mining systems effectively and efficiently.Computer sciencellw2107, gek1, dw2115, rna1, agb6, as2933, hd2200, mp2807Computer Science, Center for Computational Learning Systems, Earth and Environmental SciencesTechnical reportsImplementing Activity Structures Process Modeling On Top Of The MARVEL Environment Kernelhttp://academiccommons.columbia.edu/catalog/ac:145401
Kaiser, Gail E.; Ben-Shaul, Israel Z.; Popovich, Steven S.http://hdl.handle.net/10022/AC:P:12829Sat, 17 Mar 2012 00:00:00 +0000Our goal was to implement the activity structures model defined by Software Design & Analysis on top of the MARVEL environment kernel. This involved further design of the activity structures process definition language and enaction model as well as translation and run-time support in terms of facilities provided by MARVEL. The result is an elegant declarative control language for multi-user software processes, with data and activities defined as classes and rules in the previously existing MARVEL Strategy Language. Semantics-based concurrency control is provided by a combination of the MARVEL kernel’s lock and transaction managers and the send/receive synchronization primitives of the activity structures model.Computer sciencegek1Computer ScienceTechnical reportsAn Architecture for Dynamic Reconfiguration in a Distributed Object-Based Programming Languagehttp://academiccommons.columbia.edu/catalog/ac:145407
Hailpern, Brent; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:12831Sat, 17 Mar 2012 00:00:00 +0000Distributed applications ideally allow reconfiguration while the application is running, but changes are usually limited to adding new client and server processes and changing the bindings among such processes. In some application domains, such as real-time financial services, it is necessary to support finer grained reconfiguration at the level of entities smaller than processes, but for performance reasons it is desirable to avoid conventional approaches that require dynamic storage allocation. We present a scheme for special cases of fine-grained dynamic reconfiguration sufficient for our application domain and show how it can be used for practical changes. We introduce new language concepts to apply this scheme in the context of an object-based programming language that supports shared data in a distributed environment.Computer sciencegek1Computer ScienceTechnical reportsRule Chaining in MARVEL: Dynamic Binding of Parametershttp://academiccommons.columbia.edu/catalog/ac:145386
Heineman, George T.; Kaiser, Gail E.; Barghouti, Naser S.; Ben-Shaul, Israel Z.http://hdl.handle.net/10022/AC:P:12825Sat, 17 Mar 2012 00:00:00 +0000Marvel is a rule-based development environment (RBDE) that assists in the development of software projects. Marvel encapsulates each software development activity in a rule that specifies the condition for invoking the activity and its effects on the components of the project under development. These components are abstracted as objects and stored in a persistent object database. Each rule applies to a specific class of objects, which is specified as the parameter of the rule. Firing a rule entails binding its formal parameter to a specific object. If the rule changes the object in such a way that the conditions of other rules become satisfied, these other rules are automatically fired. A problem arises in this forward chaining model when the classes of the objects manipulated by the rules are different. Marvel has to determine which object to bind to the parameter of each rule in the chain, based on the object manipulated by the original rule that initiated the chain. We describe a heuristic approach for solving this problem in the current MARVEL implementation and introduce an algorithm approach that does better.Computer sciencegek1Computer ScienceTechnical reportsModeling Regular and Safe Concurrent Processes (Technical Summary)http://academiccommons.columbia.edu/catalog/ac:145207
Hseush, Wenwey; Balraj, Timothy S.; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:12766Thu, 08 Mar 2012 00:00:00 +0000The authors have previously described the use of data path expressions and predecessor automata in debugging concurrent systems. In this paper we examine the relationship of these models to two traditional models of concurrent processes: pomset languages and k-safe Petri net systems. We explore the regularity and safety of the concurrent languages described by each of the four models. Our main result is the equivalence of regular safe pomset languages and the languages described by safe data path expressions, safe predecessor automata and k-safe Petri net systems.Computer sciencegek1Computer ScienceTechnical reportsConsistency and Automation in Multi-User Rule-Based Development Environmentshttp://academiccommons.columbia.edu/catalog/ac:145229
Barghouti, Naser S.; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:12772Thu, 08 Mar 2012 00:00:00 +0000We investigate the scaling up of a class of single-user software development environments, which we call rule-based development environments (RBDEs), to support multiple developers cooperating together on a project. RBDEs model the software development process in terms of rules that encapsulate activities, and execute forward and backward chaining on the rules to provide assistance in carrying out the development process. There is a spectrum of assistance models, ranging from pure automation to strict consistency preservation. We describe three problems whose solutions are dependent on the choice of assistance model: (1) multiple views; (2) evolution; and (3) concurrency control. We discuss how the two extremes of the spectrum restrict the possible approaches to multiple views and evolution. In order to explore different aspects of the concurrency control problem across multiple points on the spectrum of RBDEs, we develop a maximalist assistance model and propose an approach to synchronization of cooperating developers within the context of this model.Computer sciencegek1Computer ScienceTechnical reportsAn Information Retrieval Approach for Automatically Constructing Software Librarieshttp://academiccommons.columbia.edu/catalog/ac:145232
Maarek, Yoelle S.; Berry, Daniel M.; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:12773Thu, 08 Mar 2012 00:00:00 +0000Although software reuse presents clear advantages for programmer productivity and code reliability, it is not practiced enough. One of the reasons for the only moderate success of reuse is the lack of software libraries that facilitate the actual locating and understanding of reusable components. This paper describes a technology for automatically assembling large software libraries which promote software reuse by helping the user locate the components closest to her/his needs. Software libraries are automatically assembled from a set of unorganized components by using information retrieval techniques. The construction of the library is done in two steps. First, attributes are automatically extracted from natural language documentation by using a new indexing scheme based on the notions of lexical affinities and quantity of information. Then a hierarchy for browsing is automatically generated using a clustering technique which draws only on the information provided by the attributes. Thanks to the free-text indexing scheme, tools following this approach can accept free-style natural language queries. This technology has been implemented in the GURU system, which has been applied to construct an organized library of AIX utilities. An experiment was conducted in order to evaluate the retrieval effectiveness of GURU as compared to INFOEXPLORER a hypertext library system for AIX 3 on the IBM RISC System/6000 series. We followed the usual evaluation procedure used in information retrieval, based upon recall and precision measures, and determined that our system performs 15% better on a random test set, while being much less expensive to build than INFOEXPLORER.Computer sciencegek1Computer ScienceTechnical reportsAn Object-Based Approach to Implementing Distributed Concurrency Controlhttp://academiccommons.columbia.edu/catalog/ac:145238
Popovich, Steven S.; Kaiser, Gail E.; Wu, Shyhtsun Felixhttp://hdl.handle.net/10022/AC:P:12776Thu, 08 Mar 2012 00:00:00 +0000We have added distributed concurrency control to the MELD object system by representing in progress transactions as simulated objects. Transaction objects exploit MELD’s normal message passing facilities to support the concurrency control mechanism. We have completed the implementation of an optimistic mechanism using transaction objects and have designed a two phase locking mechanism based on the same paradigm. We discuss the tradeoffs made and lessons learned, dealing both with transactions on objects and with transactions as objects.Computer sciencegek1Computer ScienceTechnical reportsAn Interoperability Model for Process-Centered Software Engineering Environments and Its Implementation in Ozhttp://academiccommons.columbia.edu/catalog/ac:144417
Ben-Shaul, Israel Z.; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:12533Fri, 10 Feb 2012 00:00:00 +0000A process-centered software engineering environment (PSEE) enables to model, evolve, and enact the process of software development and maintenance. This paper addresses the problem of process-interoperability among decentralized and autonomous PSEEs by presenting the generic International Alliance model, which consists of two elements, namely Treaty and Summit. The Treaty abstraction allows pairwise peer-peer definition of multi-site shared sub-processes that are integrated inside each of the participating sites, while retaining the definition- and evolution-autonomy of non-shared local sub-processes. Summits are the execution abstraction for Treaty-defined sub-processes. They enact Treaty sub-processes in multiple sites by successively alternating between shared and private execution modes: the former is used for the synchronous execution of the shared activities, and the latter is used for the autonomous execution of any private subtasks emanating from the shared activities. We describe the realization of the models in the Oz multi-site PSEE and evaluate the models and system based on experience gained from using Oz for production purposes. We also consider the application of the model to Petri net-based and grammar-based PSEEs.Computer sciencegek1Computer ScienceTechnical reportsEnveloping Sophisticated Tools into Process-Centered Environmentshttp://academiccommons.columbia.edu/catalog/ac:144408
Valetto, Giuseppe; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:12530Fri, 10 Feb 2012 00:00:00 +0000We present a tool integration strategy based on enveloping pre-existing tools without source code modifications or recompilation, and without assuming an extension language, application programming interface, or any other special capabilities on the part of the tool. This Black Box enveloping (or wrapping) idea has existed for a long time, but was previously restricted to relatively simple tools. We describe the design and implementation of, and experimentation with, a new Black Box enveloping facility intended for sophisticated tools --- with particular concern for the emerging class of groupware applications.Computer sciencegek1Computer ScienceTechnical reportsThe Cord Approach to Extensible Concurrency Controlhttp://academiccommons.columbia.edu/catalog/ac:144411
Heineman, George T.; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:12531Fri, 10 Feb 2012 00:00:00 +0000Database management systems (DBMSs) have been increasingly used for advanced application domains, such as software development environments, workflow management systems, computer-aided design and manufacturing, and managed healthcare. In these domains, the standard correctness model of serializability is often too restrictive. The authors introduce the notion of a concurrency control language (CCL) that allows a database application designer to specify concurrency control policies to tailor the behavior of a transaction manager. A well-crafted set of policies defines an extended transaction model. The necessary semantic information required by the CCL run-time engine is extracted from a task manager, a (logical) module by definition included in all advanced applications. This module stores task models that encode the semantic information about the transactions submitted to the DBMS. They have designed a rule-based CCL, called CORD, and have implemented a run-time engine that can be hooked to a conventional transaction manager to implement the sophisticated concurrency control required by advanced database applications. They present an architecture for systems based on CORD and describe how they integrated the CORD engine with the Exodus Storage Manager to implement altruistic locking.Computer sciencegek1Computer ScienceTechnical reportsInterfacing Oz with the PCTE OMShttp://academiccommons.columbia.edu/catalog/ac:144233
Lee, Wenke; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:12506Tue, 07 Feb 2012 00:00:00 +0000This paper details our experiment interfacing Oz with the Object Management System (OMS) of PCTE. Oz is a process-centered multi-user software development environment. PCTE is a specification which defines a language independent interface providing support mechanisms for software engineering environments (SEE) populated with CASE tools. Oz is, in theory, a SEE that can be built (or extended) using the services provided by PCTE. Oz historically has had a native OMS component whereas the PCTE OMS is an open data repository with an API for external software tools. Our experiment focused on changing Oz to use the PCTE OMS. This paper describes how several Oz components were changed in order to make the Oz server interface with the PCTE OMS. The resulting system of our experiment is an environment that has process control and integration services provided by Oz, data integration services provided by PCTE, and tool integration services provided by both. We discusses in depth the concurrency control problems that arise in such an environment and their solutions. The PCTE implementation used in our experiment is the Emeraude PCTE V 12.5.1 supplied by Transtar Software Incorporation.Computer sciencegek1Computer ScienceTechnical reportsIntegrating Groupware Activities into Workflow Management Systemshttp://academiccommons.columbia.edu/catalog/ac:144103
Ben-Shaul, Israel Z.; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:12465Fri, 03 Feb 2012 00:00:00 +0000Computer supported cooperative work (CSCW) has been recognized as a crucial enabling technology for multi-user computer-based systems, particularly in cases where synchronous human-human interaction is required between geographically dispersed users. Workflow is an emerging technology that supports complex business processes in modern corporations by allowing to explicitly define the process, and by supporting its execution in a workflow management system (WFMS). Since workflow inherently involves humans carrying out parts of the process, it is only natural to explore how to synergize these two technologies. We analyze the relationships between groupware and workflow management, present our general approach to integrating synchronous groupware tools into a WFMS, and conclude with an example process that was implemented in the Oz WFMS and integrated such tools. Our main contribution lies in the integration and synchronization of individual groupware activities into modeled workflow processes, as opposed to being a built-in part of the workflow WFMS.Computer sciencegek1Computer ScienceTechnical reportsAn Architecture for Integrating Concurrency Control into Environment Frameworkshttp://academiccommons.columbia.edu/catalog/ac:144064
Heineman, George T.; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:12442Fri, 03 Feb 2012 00:00:00 +0000Research in layered and componentized systems shows the benefit of dividing the responsibility of services into separate components. It is still an unresolved issue, however, how a system can be created from a set of existing (independently developed) components. This issue of integration is of immense concern to software architects since a proper solution would reduce duplicate implementation efforts and promote component reuse. In this paper we take a step towards this goal within the domain of software development environments (SDEs) by showing how to integrate an external concurrency control component, called Pern, with environment frameworks. We discuss two experiments where we integrated Pern with Oz, a multi-site, decentralized process centered environment, and Process WEAVER, a commercial process server. We introduce an architecture for retrofitting an external concurrency control component into an environment.Computer sciencegek1Computer ScienceTechnical reportsEnveloping Sophisticated Tools into Computer-Aided Software Engineering Environmentshttp://academiccommons.columbia.edu/catalog/ac:144079
Valetto, Giuseppe; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:12450Fri, 03 Feb 2012 00:00:00 +0000We present a CASE-tool integration strategy based on enveloping pre-existing tools without source access or assuming an API or any other special capabilities on the part of the tool. This Black Box enveloping (or wrapping) idea has been around for a long time, but was previously restricted to relatively simple tools such as compilers. We describe the design and implementation of a new Black Box enveloping facility intended for sophisticated tools-often with graphical user interfaces-with particular concern for the emerging class of groupware applications.Computer sciencegek1Computer ScienceTechnical reportsIntegrating a Transaction Manager Component with Process Weaverhttp://academiccommons.columbia.edu/catalog/ac:143948
Heineman, George T.; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:12426Thu, 02 Feb 2012 00:00:00 +0000This paper details our experience integrating a transaction manager component, called PERN with Process Weaver. Process Weaver’s Petri-net based approach is excellent for explicitly modeling concurrent activities of cooperating agents, but there is no underlying mechanism for treating conflicting actions of concurrent, independent agents. In addition, there is a need for advanced transaction support if we are to extend petri nets to use object management systems to store and access data. This paper shows several experiments we performed and our resulting implementation.Computer sciencegek1Computer ScienceTechnical reportsIncremental Process Support for Code Reengineeringhttp://academiccommons.columbia.edu/catalog/ac:143790
Heineman, George T.; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:12384Fri, 27 Jan 2012 00:00:00 +0000Reengineering a large code base can be a monumental task, and the situation becomes even worse if the code is concomitantly being modified. For the past two years, we have been using the Marvel process centered environment (PCE) for all of our software development and are currently using it to develop the Oz PCE (Marvel's successor). Towards this effort, we are reengineering Oz's code base to isolate the process engine, transaction manager, and object management system as separate components that can be mixed and matched in arbitrary systems. In this paper, we show how a PCE can guide and assist teams of users in carrying out code reengineering while allowing them to continue their normal code development. The key features to this approach are its incremental nature and the ability of the PCE to automate most of the tasks necessary to maintain the consistency of the code base.Computer sciencegek1Computer ScienceTechnical reportsA Configuration Process for a Distributed Software Development Environmenthttp://academiccommons.columbia.edu/catalog/ac:143673
Ben-Shaul, Israel Z.; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:12348Fri, 27 Jan 2012 00:00:00 +0000The authors describe work-in-progress on a configuration facility for a multi-site software development environment. The environment supports collaboration among geographically-dispersed teams of software developers. Addition and deletion of local subenvironment sites to a global environment is performed interactively inside any one of the existing local subenvironments, with the same user interface normally employed for invoking software development tools. This registration process is defined and executed using the same notation and mechanisms, respectively, as for the software development process. Everything described by the authors has been implemented and is working, but since they are in the midst of experimentation, they do not expect that the “final” system will be exactly as described.Computer sciencegek1Computer ScienceTechnical reportsDisconnected Operation in a Multi-User Software Development Environmenthttp://academiccommons.columbia.edu/catalog/ac:143682
Skopp, Peter D.; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:12351Fri, 27 Jan 2012 00:00:00 +0000Software Development Environments have traditionally relied upon a central project database and file repository, accessible to a programmer's workstation via a local area network connection. The introduction of powerful mobile computers has demonstrated the need for a new model, which allows for machines with transient network connectivity to assist programmers in product development. We propose a process-based checkout model by which process and product file that may be needed during a planned period of disconnectivity are pre-fetched with minimal user effort. Rather than selecting each file by hand, which is tedious and error-prone, the user only informs the environment of the portion of the software development process intended to be executed while disconnected. The environment is then responsible for pre-fetching the necessary file. We hope that this approach will enable programmers to continue working on a project without network access.Computer sciencegek1Computer ScienceTechnical reportsAI Techniques in Software Engineeringhttp://academiccommons.columbia.edu/catalog/ac:143576
Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:12300Fri, 20 Jan 2012 00:00:00 +0000The idea of using artificial intelligence techniques to support programming has been around for a long time. The earliest notion was to avoid programming entirely. The human user would just tell the computer what to do, without saying how to do it, and the computer would do the right thing. Even if this were feasible, however, it would be much too tedious, since each time the user would have to repeat the details of what he wanted done. So the goal of programming was to explain things to the computer only once, and then later on be able to tell the computer to do the same thing again in some short form, such as the name of the "program." Thus the idea evolved that a user would somehow tell the computer what program was desired, and the computer would write down the program in some internal form so that it could be remembered and repeated later. The assumption was that the resulting program would be correct, complete, efficient, easy to use, and so forth. It would also be exactly what the human user wanted.Computer sciencegek1Computer ScienceTechnical reportsMELDC: A Reflective Object-Oriented Coordination Languagehttp://academiccommons.columbia.edu/catalog/ac:143579
Kaiser, Gail E.; Hseushy, Wenwey; Leez, James C.; Wu, Shyhtsun Felix; Woo, Esther; Hilsdale, Eric; Meyer, Scotthttp://hdl.handle.net/10022/AC:P:12301Fri, 20 Jan 2012 00:00:00 +0000A coordination language, MELDC, for open systems programming is presented. MELDC is a C-based, concurrent, distributed object-oriented language built on a reflective architecture. Unlike other language research, the focus of MELDC is not only to study what specific language features should be designed for solving certain open system problems but also to provide programmers a high-level and efficient way to construct new features without modifying the language internals. The key to the reflective feature is the metaclass that supports shadow objects to implement secondary behaviors of objects. Thus, the behavior of an object can be extended by dynamically composing multiple secondary behaviors with the object’s primary behavior defined in the class. In this paper, both the MELDC programming model and the reflective architecture are described. Then, we introduce the mechanism of dynamic composition as well as its application in building distributed and persistent systems. In particular, a soft real-time network management system, MELDNET, is built on top of MELDC to monitor the EtherNet performance. Finally, the current status of MELDC is given.Computer sciencegek1Computer ScienceTechnical reportsExtending Attribute Grammars to Support Programming-in-the-Largehttp://academiccommons.columbia.edu/catalog/ac:143582
Micallef, Josephine; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:12302Fri, 20 Jan 2012 00:00:00 +0000Attribute grammars add specification of static semantic properties to context-free grammars, which in turn describe the syntactic structure of program units. However, context-free grammars cannot express programming-in-the-large features common in modern programming languages, including unordered collections of units, included units and sharing of included units. We present extensions to context-free grammars, and corresponding extensions to attribute grammars, suitable for defining such features. We explain how batch and incremental attribute evaluation algorithms can be adapted to support these extensions, resulting in a uniform approach to intra-unit and inter-unit static semantic analysis and translation of multi-unit programs.Computer sciencegek1Computer ScienceTechnical reportsAn Object Model for Shared Datahttp://academiccommons.columbia.edu/catalog/ac:143573
Kaiser, Gail E.; Hailpern, Brenthttp://hdl.handle.net/10022/AC:P:12299Fri, 20 Jan 2012 00:00:00 +0000The classical object model supports private data within objects and clean interfaces among objects, and by definition does not permit sharing of data among arbitrary objects. This is a problem for certain real-world applications, where the same data logically belongs 10 multiple objects and may be distributed over multiple nodes on the network. Rather than give up the advantages of encapsulated objects in modeling real-world entities, we propose a new object model that supports shared data in a distributed environment. The key is separating distribution of computation units from information hiding concerns. We introduce our new object model, describe a motivating example from the financial services domain, and then present a new language, PROFIT, based on the model.Computer sciencegek1Computer ScienceTechnical reportsCooperative Transactions for Multi-User Environmentshttp://academiccommons.columbia.edu/catalog/ac:143594
Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:12306Fri, 20 Jan 2012 00:00:00 +0000This chapter surveys extended transaction models proposed to support long duration, interactive and/or cooperative activities in the context of multi-user software development and CAD/CAM environments. Many of these are variants of the checkout model, which addresses the long duration and interactive nature of the activities supported by environments but still isolates environment users, making it difficult for them to collaborate while their activities are in progress. However, a few cooperative transaction models have been proposed to facilitate collaboration, usually while maintaining some guarantees of consistency.Computer sciencegek1Computer ScienceTechnical reportsExtending the MERCURY System to Support Teams of Ada Programmershttp://academiccommons.columbia.edu/catalog/ac:143554
Micallef, Josephine; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:12287Fri, 20 Jan 2012 00:00:00 +0000We present a model-based vision system that automatically plans the placement and optical settings of vision sensors in order to meet certain generic task requirements common to most industrial machine vision applications. From the planned viewpoints, features of interest on an object will satisfy particular constraints in the image. In this work, the vision sensor is a CCD camera equipped with a programmable lens (i.e. zoom lens) and the image constraints considered are: visibility, resolution and field of view. The proposed approach uses a geometric model of the object as well as a model of the sensor. in order to reason about the task and the environment The sensor planning system then computes the regions in space as well as the optical settings that satisfy each of the constraints separately. These results are finally combined to generate acceptable viewing locations and optical settings satisfying all constraints simultaneously. Camera planning experiments are described in which a robot-arm positions the camera at a computed location and the planned optical settings are set automatically. The corresponding scenes from the candidate viewpoints are shown demonstrating that the constraints are indeed satisfied. Other constraints, such as depth of focus, as well as other vision sensors can also be considered resulting in a fully integrated sensor planning system.Computer sciencegek1Computer ScienceTechnical reportsExtending a Tool Integration Languagehttp://academiccommons.columbia.edu/catalog/ac:143257
Gisi, Mark A.; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:12197Thu, 12 Jan 2012 00:00:00 +0000The Marvel environment supports rule-based automation of software processes. Marvel invokes external tools to carry out steps in a software process. One of the major objectives of this research is to invoke external tools to carry out steps in a software process without modifying the tools. This is achieved by encapsulating tools in envelopes, designed to abstract the details of a tool from the Marvel kernel, thereby providing a "black box" interface. Initially we used the Unix shell script language to write envelopes. However, due to several limitations of the shell language the black box abstraction could not be fully supported. We describe these limitations and discuss how we extended the shell language to obtain a new envelope language that fully supports the black box abstraction.Computer sciencegek1Computer ScienceTechnical reportsA Large-Scale, Longitudinal Study of Player Achievements in World of Warcrafthttp://academiccommons.columbia.edu/catalog/ac:143157
Bell, Jonathan Schaffer; Sheth, Swapneel Kalpesh; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:12172Wed, 11 Jan 2012 00:00:00 +0000We present a survey of usage of the popular Massively Multiplayer Online Role Playing Game, World of Warcraft. By mining publicly available data, we collected a dataset consisting of the player history for approximately six million characters, with partial data for another six million characters. This paper focuses on player achievement data in particular, exposing trends in play from this highly successful game. From this data, we present several findings on players' play styles. We correlate achievements with motivations based upon a previously-defined motivation model, and then classify players based on the categories of achievements that they pursued. Experiments show players who fall within each of these buckets can play differently, and that as players progress through game content, their play style evolves as well.Computer sciencejsb2125, sks2142, gek1Computer ScienceTechnical reportsData Quality Assurance and Performance Measurement of Data Mining for Preventive Maintenance of Power Gridhttp://academiccommons.columbia.edu/catalog/ac:143163
Wu, Leon Li; Kaiser, Gail E.; Rudin, Cynthia; Anderson, Roger N.http://hdl.handle.net/10022/AC:P:12174Wed, 11 Jan 2012 00:00:00 +0000Ensuring reliability as the electrical grid morphs into the "smart grid" will require innovations in how we assess the state of the grid, for the purpose of proactive maintenance, rather than reactive maintenance; in the future, we will not only react to failures, but also try to anticipate and avoid them using predictive modeling (machine learning and data mining) techniques. To help in meeting this challenge, we present the Neutral Online Visualization-aided Autonomic evaluation framework (NOVA) for evaluating machine learning and data mining algorithms for preventive maintenance on the electrical grid. NOVA has three stages provided through a unified user interface: evaluation of input data quality, evaluation of machine learning and data mining results, and evaluation of the reliability improvement of the power grid. A prototype version of NOVA has been deployed for the power grid in New York City, and it is able to evaluate machine learning and data mining systems effectively and efficiently.Computer sciencellw2107, gek1, rna1Computer Science, Center for Computational Learning Systems, Earth and Environmental SciencesTechnical reportsExperience with Process Modeling in the Marvel Software Development Environment Kernelhttp://academiccommons.columbia.edu/catalog/ac:142961
Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:12113Fri, 23 Dec 2011 00:00:00 +0000We have been working for several years on rule-based process modeling and the implementation of such models as part of the foundation for software development environments. We have defined a kernel, called MARVEL, for such an architecture and implemented several successive versions of the kernel and several small environments using the kernel. We have evaluated our results to date, and discovered several significant flaws and delineated several important open problems. Although the details are specific to rule-based process modeling, we believe that our insights will be valuable to other researchers and developers contemplating process modeling mechanisms.Computer sciencegek1Computer ScienceTechnical reportsA Flexible Transaction Model for Software Engineeringhttp://academiccommons.columbia.edu/catalog/ac:142958
Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:12112Fri, 23 Dec 2011 00:00:00 +0000It is generally recognized that the classical transaction model, providing atomicity and serializability, is too strong for certain application areas since it unnecessarily restricts concurrency. We are concerned with supporting cooperative work in multi-user design environments, particularly teams of programmers cooperating to develop and maintain software systems. We present an extended transaction model that meets the special requirements of software engineering projects, describe possible implementation techniques, and discuss a number of issues regarding the incorporation of such a model into multi-user software development environments.Computer sciencegek1Computer ScienceTechnical reportsModeling Concurrency in Parallel Debugginghttp://academiccommons.columbia.edu/catalog/ac:142993
Hseush, Wenwey; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:12125Fri, 23 Dec 2011 00:00:00 +0000We propose a description language, Data Path Expressions (DPEs), for modeling the behavior of parallel programs. We have designed DPEs as a high-level debugging language, where the debugging paradigm is for the programmer to describe the expected program behavior and for the debugger to compare the actual program behavior during execution to detect program errors. We classify DPEs into five subclasses according to syntactic criteria, and characterize their semantics in terms of a hierarchy of extended Petri Net models. The characterization demonstrates the power of DPEs for modeling (true) concurrency. We also present predecessor automata as a mechanism for implementing the third subclass of DPEs, which expresses bounded parallelism. Predecessor automata extend finite state automata to recognize or generate partial ordering graphs as well as strings, and provide efficient event recognizers for parallel debugging. We briefly describe the application of DPEs race conditions, deadlock and starvation.Computer sciencegek1Computer ScienceTechnical reportsTesting Reliable Distributed Applications Through Simulated Eventshttp://academiccommons.columbia.edu/catalog/ac:142895
Winfrey, Travis L.; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:12094Thu, 22 Dec 2011 00:00:00 +0000There are many distributed applications that incorporate application-specific reliability algorithms which operate on top of general purpose networking, operating system and programming language facilities. We present a framework for application-level reliability testing suitable for a wide range of distributed applications, and describe how we've applied it to one particular application, Mercury, a distributed, multi-user programming environment.Computer sciencegek1Computer ScienceTechnical reportsObject-Oriented Programming Language Facilities for Concurrency Controlhttp://academiccommons.columbia.edu/catalog/ac:142930
Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:12105Thu, 22 Dec 2011 00:00:00 +0000Concurrent object-oriented programming systems require support for concurrency control, to enforce consistent commitment of changes and to support program-initiated rollback after application-specific failures. We have explored three different concurrency control models -- atomic blocks, serializable transactions, and commit-serializable transactions -- as part of the MELD programming language. We present our designs, discuss certain programming problems and implementation issues, and compare our work on MELD to other concurrent object-based systems.Computer sciencegek1Computer ScienceTechnical reportsMELDing Multiple Granularities of Parallelismhttp://academiccommons.columbia.edu/catalog/ac:142933
Kaiser, Gail E.; Popovich, Steven S.; Hseush, Wenwey; Wu, Shyhtsun Felixhttp://hdl.handle.net/10022/AC:P:12106Thu, 22 Dec 2011 00:00:00 +0000We are developing an experimental programming language, MELD, that supports a range of concurrent styles by supporting multiple programming paradigms at multiple levels of granularity. MELD integrates three granularities of parallelism: macro dataflow among statements within a method or among methods for fine grain concurrency, synchronous or asynchronous message passing among local or remote objects for medium grain concurrency, and transactions for large grain concurrency among users.Computer sciencegek1Computer ScienceTechnical reportsAutomated Tutoring in Interactive Environments: A Task Centered Approachhttp://academiccommons.columbia.edu/catalog/ac:142820
Wolz, Ursula; McKeown, Kathleen; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:12068Wed, 21 Dec 2011 00:00:00 +0000Tutoring in interactive computing environments is sometimes more properly understood as consulting. A tutor's implied curriculum may be adaptable to the user's knowledge and experience but still not meet the user's immediate needs -- to get some task done. A consultant however, can dynamically adapt to address the task at hand. We present a user's goal-centered approach to tutoring in interactive environments, and describe how we automate certain tutoring strategies appropriate for consulting behavior. We have implemented our approach in GENIE, a question answering system for the Berkeley Unix Mail system. We focus on the pedagogical strategies employed by GENIE to best meet the user's immediate needs.Computer sciencekrm8, gek1Computer ScienceTechnical reportsAn Automated Consultant for Interactive Environmentshttp://academiccommons.columbia.edu/catalog/ac:142817
Wolz, Ursula; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:12067Wed, 21 Dec 2011 00:00:00 +0000Interactive computing environments provide facilities intended to support and assist the range from novice to expert users, but casual users tend to get trapped in the starter set of commands. We have developed a rule-based technology for providing on-line assistance calibrated to both the task at hand and the user's past experience using the system. Such assistance helps users to progress to more advanced features. We present our automated consultant and describe its application to a practical domain, the Berkeley Unix mail system.Computer sciencegek1Computer ScienceTechnical reportsExtended Transaction Models for Software Development Environmentshttp://academiccommons.columbia.edu/catalog/ac:142838
Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:12075Wed, 21 Dec 2011 00:00:00 +0000This technical report consists of two papers discussing concurrency control facilities for multiuser software development environments. A Marvelous Extended Transaction Processing Model briefly sketches the previously developed commit-serializability model and then applies it to the MARVEL kernel for software development environments. A Participant Semantics for Serializable Transactions describes our first pass at a different extended transaction model that moves users inside the system, so certain users may participate in transactions and the interactions among transactions while all other users observe only a serial order for the transactions.Computer sciencegek1Computer ScienceTechnical reportsSplit-Transactions for Open-Ended Activitieshttp://academiccommons.columbia.edu/catalog/ac:142708
Pu, Calton; Kaiser, Gail E.; Hutchinson, Normanhttp://hdl.handle.net/10022/AC:P:12033Sat, 17 Dec 2011 00:00:00 +0000Open-ended activities such as CAD/Cam, VLSI layout and software development require consistent concurrent access and fault tolerance associated with database transactions, but their uncertain duration, uncertain developments during execution and long interaction with other concurrent activities break traditional transaction atomicity boundaries. We propose split-transaction as a new database operation that solves the above problems by permitting transactions to commit data that will not change. Thus an open-ended activity can release the committed data and serialize interactions with other concurrent activities through the committed data. This technical report was originally titled Support of Multiuser Software Development Environments.Computer sciencegek1Computer ScienceTechnical reportsSupport for Reliable Distributed Computinghttp://academiccommons.columbia.edu/catalog/ac:142711
Kaiser, Gail E.; Hseush, Wenweyhttp://hdl.handle.net/10022/AC:P:12034Sat, 17 Dec 2011 00:00:00 +0000This technical report consists of two papers describing support for reliable distributed computing. Data Path Debugging: Data-Oriented Debugging for a Concurrent Programming Language explains our goal of data-oriented debugging, and then presents Data Path Expression (DPE) debugging, our approach to data-oriented debugging for concurrent programming languages. DPE is being implemented as pan of MD, the MELD Debugger. A Network Architecture for Reliable Distributed Computing proposes a reliable distributed environment (RDE) based on an efficient and reliable extension to datagram communications. It gives simulation results for coupled relations based on different algorithms， node failure rate, recover rate, message sending rate and data missing rate, and to illustrate behavior of distributed systems constructed using our view section model on top of RDE.Computer sciencegek1Computer ScienceTechnical reportsGAEA Action Equations Paradigmhttp://academiccommons.columbia.edu/catalog/ac:142705
Kaiser, Gail E.; Kaplan, Simon M.http://hdl.handle.net/10022/AC:P:12032Sat, 17 Dec 2011 00:00:00 +0000This technical report consists of two papers describing the GAEA action equations paradigm. Incremental Dynamic Semantics for Language-based Programming Environments explains why attribute grammars are not suitable for expressing dynamic semantics and presents action equations, an extension of attribute grammars suitable for specifying the static and the dynamic semantics of programming languages. It describes how action equations can be used to generate language-based programming environments that incrementally derive static and dynamic properties as the user modifies and debugs the program. Rapid Prototyping of Concurrent Programming Languages extends this technology to a concurrent framework. It describes an (unimplemented) system that generates a parallel interpreter for the language and provides runtime support for the synchronization primitives and other facilities in the language.Computer sciencegek1Computer ScienceTechnical reportsA Retrospective on DOSE: An Interpretive Approach to Structure Editor Generationhttp://academiccommons.columbia.edu/catalog/ac:142714
Kaiser, Gail E.; Feiler, Peter H.; Jalili, Fahimeh; Schlichter, Johann H.http://hdl.handle.net/10022/AC:P:12035Sat, 17 Dec 2011 00:00:00 +0000DOSE is unique among structu.re editor generators in its interpretive approach. This approach leads to very fast turn-around time for changes and provides multi-language facilities for no additional effort or cost. This article compares the interpretive approach to the compilation approach of other structu.re editor generators. It describes some of the design and implementation decisions made and remade during this project and the lessons learned. It emphasizes the advantages and disadvantages of DOSE with respect to other structure editing systems.Computer sciencegek1Computer ScienceTechnical reportsSoftware Development Environments for Very Large Software Systemshttp://academiccommons.columbia.edu/catalog/ac:142071
Kaiser, Gail E.; Maarek, Yoelle S.; Perry, Dewayne E.; Schwanke, Robert W.http://hdl.handle.net/10022/AC:P:11847Mon, 28 Nov 2011 00:00:00 +0000This technical report consists of the three related papers. Living with Inconsistency in Large Systems describes CONMAN, an environment that identifies and tracks version inconsistencies, permitting debugging and testing to proceed even though the executable image contains certain non-fatal inconsistencies. The next two papers are both from the INFUSE project. Change Management for Very Large Software Systems presents the new non-Euclidean hierarchical clustering algorithm used by the INFUSE change management system to cluster modules according to the strengths of their interdependencies. Models of Software Development Environments presents a general model of software development environments consisting of three components - policies, mechanisms and structures - and classifies existing and proposed environments into the individual, family, city and state classes according to the size of projects that could be adequately supported.Computer sciencegek1Computer ScienceTechnical reportsReliable Network Communicationshttp://academiccommons.columbia.edu/catalog/ac:142068
Kaiser, Gail E.; Cycowicz, Yael J.; Hseush, Wenwey; Micallef, Josephinehttp://hdl.handle.net/10022/AC:P:11846Mon, 28 Nov 2011 00:00:00 +0000This technical report consists of three papers from the INTERCOMS project. A Network Architecture for Reliable Distributed Computing introduces the view section model, a network layer for exception handling in response to disruptions in communication channels due to failures of network links or nodes. Remote Exception Handling discusses for a network layer for exception handling among cooperating application processes. Demand-Driven Parameter Passing in Remote Procedure Call describes how remote exception handling solves the problem of passing referential data types (pointers) as parameters to remote procedures.Computer sciencegek1Computer ScienceTechnical reportsMERCURY: Distributed Incremental Attribute Grammar Evaluationhttp://academiccommons.columbia.edu/catalog/ac:142075
Kaiser, Gail E.; Micallef, Josephine; Kaplan, Simon M.http://hdl.handle.net/10022/AC:P:11848Mon, 28 Nov 2011 00:00:00 +0000This technical report consists of the two most recent papers from the MERCURY project Multiuser, Distributed Language-Based Environments explains the application of incremental attribute grammar evaluation algorithms to generation of distributed programming environments and describes the implementation of the MERCURY system. Version and Configuration Control in Distributed Language-Based Environments presents new algorithms that permit MERCURY to support multiple versions and configurations of modules and to more efficiently propagate changes to aggregate attributes.Computer sciencegek1Computer ScienceTechnical reportsMELD: A Multi-Paradigm Language with Objects, Dataflow and Moduleshttp://academiccommons.columbia.edu/catalog/ac:142078
Kaiser, Gail E.; Garlan, Davidhttp://hdl.handle.net/10022/AC:P:11849Mon, 28 Nov 2011 00:00:00 +0000This technical report consists of the two most recent papers from the MELD project. Melding Software Systems from Reusable Building Blocks describes MELD as a declarative language that combines facilities from the package library, software generation and object-oriented language approaches to reusability. MEWing Data Flow and Object-Oriented Programming emphasizes the multi-paradigm nature of MELD and introduces overriding of inherited facilities and generic features, and discusses compile-time error detection.Computer sciencegek1Computer ScienceTechnical reportsSpecification of Interpreters and Debuggers Using an Extension of Attribute Grammarshttp://academiccommons.columbia.edu/catalog/ac:141234
Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:11733Mon, 07 Nov 2011 00:00:00 +0000Recent research in programming environments has focused on the generation of software tools from specifications. Several specification mechanisms have been proposed, and the most successful of these has been attribute grammars. Attribute grammars have been successfully applied to compile-time tools such as type checkers and code generators, which perform static analysis of the program, but have hitherto seemed unsuited to the description of run-time tools such as interpreters and debuggers that involve dynamic interaction with the user of the programming environment. This report describes an extension to attribute grammars that solves this problem. The extended attribute grammars are suitable for the specification of all semantics processing performed by single-user programming environments.Computer sciencegek1Computer ScienceTechnical reportsGeneration of Distributed Programming Environmentshttp://academiccommons.columbia.edu/catalog/ac:141175
Kaiser, Gail E.; Kaplan, Simon M.; Micallef, Josephinehttp://hdl.handle.net/10022/AC:P:11676Mon, 31 Oct 2011 00:00:00 +0000This technical report consists of three related papers in the area of distributed programming environments. Incremental Attribute Evaluation in Distributed Language-Based Environments presents algorithms that extend existing technology for the generation of single-user language-based editors from attribute grammars to the cases of multiple-user concurrent and distributed environments. Multi-User Distributed Language-Based Environment, an extended abstract, provides additional information on how to apply the algorithms. Reliability in Distributed Programming Environments presents additional algorithms that extend our results to unreliable networks.Computer sciencegek1Computer ScienceTechnical reportsMELD/ Features: An Object-Oriented Approach to Reusable Softwarehttp://academiccommons.columbia.edu/catalog/ac:141178
Kaiser, Gail E.; Garlan, Davidhttp://hdl.handle.net/10022/AC:P:11678Mon, 31 Oct 2011 00:00:00 +0000This technical report consists of three related papers in the area of reusable software. Synthesis of Programming Environments from Reusable Building Blocks presents the notion of 'features' as an approach to reusable descriptions for the generation of programming environments. Composing Software Systems from Reusable Building Blocks presents MELD, a declarative language based on features, and generalizes features to the description of reusable software for general applications. MELD: A Declarative Language for Writing Methods focuses on ~1ELD's capabilities for describing the behavior of software systems.Computer sciencegek1Computer ScienceTechnical reportsSMILE/ MARVEL: Two Approaches to Knowledge-Based Programming Environmentshttp://academiccommons.columbia.edu/catalog/ac:141181
Kaiser, Gail E.; Feiler, Peter H.http://hdl.handle.net/10022/AC:P:11683Mon, 31 Oct 2011 00:00:00 +0000This technical report consists of three related papers in the area of intelligent assistance for software development and maintenance. Intelligent Assistance without Artificial Intelligence describes SMILE, a software engineering environment that assists teams of programmers without using AI technology. An Architecture for Intelligent Assistance in Software Development presents an AI approach to generalizing the capabilities of SMILE. Granularity Issues in a Knowledge-Based Programming Environment briefly describes MARVEL, an intelligent assistant based on this AI approach, and compares it to SMILE.Computer sciencegek1Computer ScienceTechnical reportsChange Management Support for Large Software Systemshttp://academiccommons.columbia.edu/catalog/ac:141172
Kaiser, Gail E.; Perry, Dewayne E.; Schwanke, Robert W.http://hdl.handle.net/10022/AC:P:11675Mon, 31 Oct 2011 00:00:00 +0000This technical report consists of three related papers in the area of management of source code changes for large software systems. Infuse: A Toolfor Automatically Managing and Coordinating Source Changes in Large Systems presents Infuse, a software engineering environment that automatically panitions the source code files to be changed in order to limit the complexity of change propagation and negotiation of conflicting changes. Workspaces and Experimental Databases: Automated Support for Cooperation Among Programmers describes Infuse in more detail, focusing on the interactions between Infuse and source code modification tools. Smarter Recompilation presents an algorithm that could be used by Infuse or other software engineering environments to reduce recompilation costs after source code changes.Computer sciencegek1Computer ScienceTechnical reportsMoney for Nothing and Privacy for Free?http://academiccommons.columbia.edu/catalog/ac:140259
Sheth, Swapneel Kalpesh; Malkin, Tal G.; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:11490Wed, 19 Oct 2011 00:00:00 +0000Privacy in the context of ubiquitous social computing systems has become a major concern for the society at large. As the number of online social computing systems that collect user data grows, this privacy threat is further exacerbated. There has been some work (both, recent and older) on addressing these privacy concerns. These approaches typically require extra computational resources, which might be beneficial where privacy is concerned, but when dealing with Green Computing and sustainability, this is not a great option. Spending more computation time results in spending more energy and more resources that make the software system less sustainable. Ideally, what we would like are techniques for designing software systems that address these privacy concerns but which are also sustainable - systems where privacy could be achieved "for free," i.e., without having to spend extra computational effort. In this paper, we describe how privacy can be achieved for free - an accidental and beneficial side effect of doing some existing computation - and what types of privacy threats it can mitigate. More precisely, we describe a "Privacy for Free" design pattern and show its feasibility, sustainability, and utility in building complex social computing systems.Computer sciencesks2142, tm2118, gek1Computer ScienceTechnical reportsSecret Ninja Testing with HALO Software Engineeringhttp://academiccommons.columbia.edu/catalog/ac:135498
Bell, Jonathan Schaffer; Sheth, Swapneel Kalpesh; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:10684Mon, 11 Jul 2011 00:00:00 +0000Software testing traditionally receives little attention in early computer science courses. However, we believe that if exposed to testing early, students will develop positive habits for future work. As we have found that students typically are not keen on testing, we propose an engaging and socially-oriented approach to teaching software testing in introductory and intermediate computer science courses. Our proposal leverages the power of gaming utilizing our previously described system HALO. Unlike many previous approaches, we aim to present software testing in disguise - so that students do not recognize (at first) that they are being exposed to software testing. We describe how HALO could be integrated into course assignments as well as the benefits that HALO creates.Computer sciencejsb2125, sks2142, gek1Computer ScienceTechnical reportsTowards Diversity in Recommendations Using Social Networkshttp://academiccommons.columbia.edu/catalog/ac:135477
Sheth, Swapneel Kalpesh; Bell, Jonathan Schaffer; Arora, Nipun; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:10677Mon, 11 Jul 2011 00:00:00 +0000While there has been a lot of research towards improving the accuracy of recommender systems, the resulting systems have tended to become increasingly narrow in suggestion variety. An emerging trend in recommendation systems is to actively seek out diversity in recommendations, where the aim is to provide unexpected, varied, and serendipitous recommendations to the user. Our main contribution in this paper is a new approach to diversity in recommendations called "Social Diversity," a technique that uses social network information to diversify recommendation results. Social Diversity utilizes social networks in recommender systems to leverage the diverse underlying preferences of different user communities to introduce diversity into recommendations. This form of diversification ensures that users in different social networks (who may not collaborate in real life, since they are in a different network) share information, helping to prevent siloization of knowledge and recommendations. We describe our approach and show its feasibility in providing diverse recommendations for the MovieLens dataset.Computer sciencesks2142, jsb2125, na2271, gek1Computer ScienceTechnical reportsEstimation of System Reliability Using a Semiparametric Modelhttp://academiccommons.columbia.edu/catalog/ac:135421
Wu, Leon Li; Teravainen, Timothy Kaleva; Kaiser, Gail E.; Anderson, Roger N.; Boulanger, Albert G.; Rudin, Cynthiahttp://hdl.handle.net/10022/AC:P:10670Fri, 08 Jul 2011 00:00:00 +0000An important problem in reliability engineering is to predict the failure rate, that is, the frequency with which an engineered system or component fails. This paper presents a new method of estimating failure rate using a semiparametric model with Gaussian process smoothing. The method is able to provide accurate estimation based on historical data and it does not make strong a priori assumptions of failure rate pattern (e.g., constant or monotonic). Our experiments of applying this method in power system failure data compared with other models show its efficacy and accuracy. This method can be used in estimating reliability for many other systems, such as software systems or components.Computer sciencellw2107, tkt2103, gek1, rna1Statistics, Computer Science, Center for Computational Learning SystemsTechnical reportsEmpirical Evaluation of Approaches to Testing Applications without Test Oracleshttp://academiccommons.columbia.edu/catalog/ac:133608
Murphy, Christian; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:10525Thu, 09 Jun 2011 00:00:00 +0000Software testing of applications in fields like scientific computing, simulation, machine learning, etc. is particularly challenging because many applications in these domains have no reliable "test oracle" to indicate whether the program's output is correct when given arbitrary input. A common approach to testing such applications has been to use a "pseudo-oracle", in which multiple independently-developed implementations of an algorithm process an input and the results are compared. Other approaches include the use of program invariants, formal specification languages, trace and log file analysis, and metamorphic testing. In this paper, we present the results of two empirical studies in which we compare the effectiveness of some of these approaches, including metamorphic testing, pseudo-oracles, and runtime assertion checking. We also analyze the results in terms of the software development process, and discuss suggestions for practitioners and researchers who need to test software without a test oracle.Computer sciencegek1Computer ScienceTechnical reportsHALO (Highly Addictive, sociaLly Optimized) Software Engineeringhttp://academiccommons.columbia.edu/catalog/ac:133639
Sheth, Swapneel Kalpesh; Bell, Jonathan Schaffer; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:10533Thu, 09 Jun 2011 00:00:00 +0000In recent years, computer games have become increasingly social and collaborative in nature. Massively multiplayer online games, in which a large number of players collaborate with each other to achieve common goals in the game, have become extremely pervasive. By working together towards a common goal, players become more engrossed in the game. In everyday work environments, this sort of engagement would be beneficial, and is often sought out. We propose an approach to software engineering called HALO that builds upon the properties found in popular games, by turning work into a game environment. Our proposed approach can be viewed as a model for a family of prospective games that would support the software development process. Utilizing operant conditioning and flow theory, we create an immersive software development environment conducive to increased productivity. We describe the mechanics of HALO and how it could fit into typical software engineering processes.Computer sciencesks2142, jsb2125, gek1Computer ScienceTechnical reportsEmpirical Study of Concurrency Mutation Operators for Javahttp://academiccommons.columbia.edu/catalog/ac:133611
Wu, Leon Li; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:10526Thu, 09 Jun 2011 00:00:00 +0000Mutation testing is a white-box fault-based software testing technique that applies mutation operators to modify program source code or byte code in small ways and then runs these modified programs (i.e., mutants) against a test suite in order to measure its effectiveness and locate the weaknesses either in the test data or in the program that are seldom or never exposed during normal execution. In this paper, we describe our implementation of a generic mutation testing framework and the results of applying three sets of concurrency mutation operators on four example Java programs through empirical study and analysis.Computer sciencellw2107, gek1Computer Science, Center for Computational Learning SystemsTechnical reportsThe Tradeoffs of Societal Computinghttp://academiccommons.columbia.edu/catalog/ac:133596
Sheth, Swapneel Kalpesh; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:10521Thu, 09 Jun 2011 00:00:00 +0000As Social Computing has increasingly captivated the general public, it has become a popular research area for computer scientists. Social Computing research focuses on online social behavior and using artifacts derived from it for providing recommendations and other useful community knowledge. Unfortunately, some of that behavior and knowledge incur societal costs, particularly with regards to Privacy, which is viewed quite differently by different populations as well as regulated differently in different locales. But clever technical solutions to those challenges may impose additional societal costs, e.g., by consuming substantial resources at odds with Green Computing, another major area of societal concern. We propose a new crosscutting research area, Societal Computing, that focuses on the technical tradeoffs among computational models and application domains that raise significant societal issues. We highlight some of the relevant research topics and open problems that we foresee in Societal Computing where software engineering research approaches and techniques seem particularly likely to be fruitful. We feel that these topics, and Societal Computing in general, need to gain prominence as they will provide useful avenues of research leading to increasing benefits for society as a whole.Computer sciencesks2142, gek1Computer ScienceTechnical reportsTowards Using Cached Data Mining for Large Scale Recommender Systemshttp://academiccommons.columbia.edu/catalog/ac:133590
Sheth, Swapneel Kalpesh; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:10519Thu, 09 Jun 2011 00:00:00 +0000Recommender systems are becoming increasingly popular. As these systems become commonplace and the number of users increases, it will become important for these systems to be able to cope with a large and diverse set of users whose recommendation needs may be very different from each other. In particular, large scale recommender systems will need to ensure that users' requests for recommendations can be answered with low response times and high throughput. In this paper, we explore how to use caches and cached data mining to improve the performance of recommender systems by improving throughput and reducing response time for providing recommendations. We describe the structure of our cache, which can be viewed as a prefetch cache that prefetches all types of supported recommendations, and how it is used in our recommender system. We also describe the results of our simulation experiments to measure the efficacy of our cache.Computer sciencesks2142, gek1Computer ScienceTechnical reportsBaseline: Metrics for setting a baseline for web vulnerability scannershttp://academiccommons.columbia.edu/catalog/ac:133584
Dai, Huning; Glass, Michael; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:10517Thu, 09 Jun 2011 00:00:00 +0000As web scanners are becoming more popular because they are faster and cheaper than security consultants, the trend of relying on these scanners also brings a great hazard: users can choose a weak or outdated scanner and trust incomplete results. Therefore, benchmarks are created to both evaluate and compare the scanners. Unfortunately, most existing benchmarks suffer from various drawbacks, often by testing against inappropriate criteria that does not reflect the user's needs. To deal with this problem, we present an approach called Baseline that coaches the user in picking the minimal set of weaknesses (i.e., a baseline) that a qualified scanner should be able to detect and also helps the user evaluate the effectiveness and efficiency of the scanner in detecting those chosen weaknesses. Baseline's goal is not to serve as a generic ranking system for web vulnerability scanners, but instead to help users choose the most appropriate scanner for their specific needs.Computer sciencehd2210, mgg2102, gek1Computer ScienceTechnical reportsAutomatic Detection of Defects in Applications without Test Oracleshttp://academiccommons.columbia.edu/catalog/ac:133587
Murphy, Christian; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:10518Thu, 09 Jun 2011 00:00:00 +0000In application domains that do not have a test oracle, such as machine learning and scientific computing, quality assurance is a challenge because it is difficult or impossible to know in advance what the correct output should be for general input. Previously, metamorphic testing has been shown to be a simple yet effective technique in detecting defects, even without an oracle. In metamorphic testing, the application's ``metamorphic properties'' are used to modify existing test case input to produce new test cases in such a manner that, when given the new input, the new output can easily be computed based on the original output. If the new output is not as expected, then a defect must exist. In practice, however, metamorphic testing can be a manually intensive technique for all but the simplest cases. The transformation of input data can be laborious for large data sets, and errors can occur in comparing the outputs when they are very complex. In this paper, we present a tool called Amsterdam that automates metamorphic testing by allowing the tester to easily set up and conduct metamorphic tests with little manual intervention, merely by specifying the properties to check, configuring the framework, and running the software. Additionally, we describe an approach called Heuristic Metamorphic Testing, which addresses issues related to false positives and non-determinism, and we present the results of new empirical studies that demonstrate the effectiveness of metamorphic testing techniques at detecting defects in real-world programs without test oracles.Computer sciencegek1Computer ScienceTechnical reportsOn Effective Testing of Health Care Simulation Softwarehttp://academiccommons.columbia.edu/catalog/ac:133635
Murphy, Christian; Raunak, M. S.; King, Andrew; Chen, Sanjien; Imbriano, Christopher; Kaiser, Gail E.; Lee, Insup; Sokolsky, Oleg; Clarke, Lori; Osterweil, Leonhttp://hdl.handle.net/10022/AC:P:10532Thu, 09 Jun 2011 00:00:00 +0000Health care professionals rely on software to simulate anatomical and physiological elements of the human body for purposes of training, prototyping, and decision making. Software can also be used to simulate medical processes and protocols to measure cost effectiveness and resource utilization. Whereas much of the software engineering research into simulation software focuses on validation (determining that the simulation accurately models real-world activity), to date there has been little investigation into the testing of simulation software itself, that is, the ability to effectively search for errors in the implementation. This is particularly challenging because often there is no test oracle to indicate whether the results of the simulation are correct. In this paper, we present an approach to systematically testing simulation software in the absence of test oracles, and evaluate the effectiveness of the technique.Computer sciencegek1Computer ScienceTechnical reportsPOWER: Parallel Optimizations With Executable Rewritinghttp://academiccommons.columbia.edu/catalog/ac:133629
Arora, Nipun; Bell, Jonathan Schaffer; Kim, Martha Allen; Singh, Vishal; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:10530Thu, 09 Jun 2011 00:00:00 +0000The hardware industry's rapid development of multicore and many core hardware has outpaced the software industry's transition from sequential to parallel programs. Most applications are still sequential, and many cores on parallel machines remain unused. We propose a tool that uses data-dependence profiling and binary rewriting to parallelize executables without access to source code. Our technique uses Bernstein's conditions to identify independent sets of basic blocks that can be executed in parallel, introducing a level of granularity between fine-grained instruction level and coarse grained task level parallelism. We analyze dynamically generated control and data dependence graphs to find independent sets of basic blocks which can be parallelized. We then propose to parallelize these candidates using binary rewriting techniques. Our technique aims to demonstrate the parallelism that remains in serial application by exposing concrete opportunities for parallelism.Computer sciencena2271, jsb2125, mak2191, gek1Computer ScienceTechnical reportsCONFU: Configuration Fuzzing Testing Framework for Software Vulnerability Detectionhttp://academiccommons.columbia.edu/catalog/ac:133544
Dai, Huning; Murphy, Christian; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:10507Wed, 08 Jun 2011 00:00:00 +0000Many software security vulnerabilities only reveal themselves under certain conditions, i.e., particular configurations and inputs together with a certain runtime environment. One approach to detecting these vulnerabilities is fuzz testing. However, typical fuzz testing makes no guarantees regarding the syntactic and semantic validity of the input, or of how much of the input space will be explored. To address these problems, we present a new testing methodology called Configuration Fuzzing. Configuration Fuzzing is a technique whereby the configuration of the running application is mutated at certain execution points, in order to check for vulnerabilities that only arise in certain conditions. As the application runs in the deployment environment, this testing technique continuously fuzzes the configuration and checks "security invariants'' that, if violated, indicate a vulnerability. We discuss the approach and introduce a prototype framework called ConFu (CONfiguration FUzzing testing framework) for implementation. We also present the results of case studies that demonstrate the approach's feasibility and evaluate its performance.Computer sciencehd2210, gek1Computer ScienceTechnical reportsComparing Speed of Provider Data Entry: Electronic Versus Paper Methodshttp://academiccommons.columbia.edu/catalog/ac:133547
Jackson, Kevin M.; Kaiser, Gail E.; Wong, Lyndon; Rabinowitz, Daniel; Chiang, Michael F.http://hdl.handle.net/10022/AC:P:10508Wed, 08 Jun 2011 00:00:00 +0000Electronic health record (EHR) systems have significant potential advantages over traditional paper-based systems, but they require that providers assume responsibility for data entry. One significant barrier to adoption of EHRs is the perception of slowed data-entry by providers. This study compares the speed of data-entry using computer-based templates vs. paper for a large eye clinic, using 10 subjects and 10 simulated clinical scenarios. Dataentry into the EHR was significantly slower (p<0.01) than traditional paper forms.Computer sciencegek1, dr105Statistics, Biomedical Informatics, Computer Science, OphthalmologyTechnical reportsThe weHelp Reference Architecture for Community-Driven Recommender Systemshttp://academiccommons.columbia.edu/catalog/ac:133550
Sheth, Swapneel Kalpesh; Arora, Nipun; Murphy, Christian; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:10509Wed, 08 Jun 2011 00:00:00 +0000Recommender systems have become increasingly popular. Most research on recommender systems has focused on recommendation algorithms. There has been relatively little research, however, in the area of generalized system architectures for recommendation systems. In this paper, we introduce weHelp - a reference architecture for social recommender systems. Our architecture is designed to be application and domain agnostic, but we briefly discuss here how it applies to recommender systems for software engineering.Computer sciencesks2142, na2271, gek1Computer ScienceTechnical reportsTesting and Validating Machine Learning Classifiers by Metamorphic Testinghttp://academiccommons.columbia.edu/catalog/ac:133529
Xie, Xiaoyuan; Ho, Joshua W. K.; Murphy, Christian; Kaiser, Gail E.; Xu, Baowen; Chen, Tsong Yuehhttp://hdl.handle.net/10022/AC:P:10501Tue, 07 Jun 2011 00:00:00 +0000Machine Learning algorithms have provided important core functionality to support solutions in many scientific computing applications - such as computational biology, computational linguistics, and others. However, it is difficult to test such applications because often there is no "test oracle" to indicate what the correct output should be for arbitrary input. To help address the quality of scientific computing software, in this paper we present a technique for testing the implementations of machine learning classification algorithms on which such scientific computing software depends. Our technique is based on an approach called "metamorphic testing", which has been shown to be effective in such cases. Also presented is a case study on a real-world machine learning application framework, and a discussion of how programmers implementing machine learning algorithms can avoid the common pitfalls discovered in our study. We also conduct mutation analysis and cross-validation, which reveal that our method has very high effectiveness in killing mutants, and that observing expected cross-validation result alone is not sufficient to test for the correctness of a supervised classification program. Metamorphic testing is strongly recommended as a complementary approach. Finally we discuss how our findings can be used in other areas of computational science and engineering.Computer sciencegek1Computer ScienceTechnical reportsAutomatic Detection of Previously-Unseen Application States for Deployment Environment Testing and Analysishttp://academiccommons.columbia.edu/catalog/ac:133532
Murphy, Christian; Vaughan, Moses; Ilahi, Waseem; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:10502Tue, 07 Jun 2011 00:00:00 +0000For large, complex software systems, it is typically impossible in terms of time and cost to reliably test the application in all possible execution states and configurations before releasing it into production. One proposed way of addressing this problem has been to continue testing and analysis of the application in the field, after it has been deployed. The theory behind this "perpetual testing" approach is that over time, defects will reveal themselves given that multiple instances of the same application may be run globally with different configurations, in different environments, under different patterns of usage, and in different system states. A practical limitation of many automated approaches to deployment environment testing and analysis is the potentially high performance overhead incurred by the necessary instrumentation. However, it may be possible to reduce this overhead by selecting test cases and performing analysis only in previously-unseen application states, thus reducing the number of redundant tests and analyses that are run. Solutions for fault detection, model checking, security testing, and fault localization in deployed software may all benefit from a technique that ignores application states that have already been tested or explored. In this paper, we apply such a technique to a testing methodology called "In Vivo Testing", which conducts tests in deployed applications, and present a solution that ensures that tests are only executed in states that the application has not previously encountered. In addition to discussing our implementation, we present the results of an empirical study that demonstrates its effectiveness, and explain how the new approach can be generalized to assist other automated testing and analysis techniques.Computer sciencemjv2123, wki2001, gek1Computer ScienceTechnical reportsA Runtime Adaptation Framework for Native C and Bytecode Applicationshttp://academiccommons.columbia.edu/catalog/ac:110509
Griffith, Rean; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:29446Thu, 05 May 2011 00:00:00 +0000The need for self-healing software to respond with a reactive, proactive or preventative action as a result of changes in its environment has added the non-functional requirement of adaptation to the list of facilities expected in self-managing systems. The adaptations we are concerned with assist with problem detection, diagnosis and remediation. Many existing computing systems do not include such adaptation mechanisms, as a result these systems either need to be re-designed to include them or there needs to be a mechanism for retro-fitting these mechanisms. The purpose of the adaptation mechanisms is to ease the job of the system administrator with respect to managing software systems. This paper introduces Kheiron, a framework for facilitating adaptations in running programs in a variety of execution environments without requiring the redesign of the application. Kheiron manipulates compiled C programs running in an unmanaged execution environment as well as programs running in Microsoft's Common Language Runtime and SunMicrosystems' Java VirtualMachine. We present case-studies and experiments that demonstrate the feasibility of using Kheiron to support self-healing systems. We also describe the concepts and techniques used to retro-fit adaptations onto existing systems in the various execution environments.Computer sciencegek1Computer ScienceTechnical reportsRAS-Models: A Building Block for Self-Healing Benchmarkshttp://academiccommons.columbia.edu/catalog/ac:146026
Griffith, Rean; Virmani, Ritika; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:29580Thu, 05 May 2011 00:00:00 +0000To evaluate the efficacy of self-healing systems a rigorous, objective, quantitative benchmarking methodology is needed. However, developing such a benchmark is a non-trivial task given the many evaluation issues to be resolved, including but not limited to: quantifying the impacts of faults, analyzing various styles of healing (reactive, preventative, proactive), accounting for partially automated healing and accounting for incomplete/imperfect healing. We posit, however,that it is possible to realize a self-healing benchmark using a collection of analytical techniques and practical tools as building blocks. This paper highlights the flexibility of one analytical tool, the Reliability, Availability and Serviceability (RAS) model, and illustrates its power and relevance to the problem of evaluating self-healing mechanisms/systems, when combined with practical tools for fault-injection.Computer sciencerg2023, rv2171, gek1Computer ScienceTechnical reportsAn Approach to Software Testing of Machine Learning Applicationshttp://academiccommons.columbia.edu/catalog/ac:110692
Murphy, Christian; Kaiser, Gail E.; Arias, Martahttp://hdl.handle.net/10022/AC:P:29502Thu, 28 Apr 2011 00:00:00 +0000Some machine learning applications are intended to learn properties of data sets where the correct answers are not already known to human users. It is challenging to test such ML software, because there is no reliable test oracle. We describe a software testing approach aimed at addressing this problem. We present our findings from testing implementations of two different ML ranking algorithms: Support Vector Machines and MartiRank.Computer sciencegek1Computer ScienceTechnical reportsBackstop: A Tool for Debugging Runtime Errorshttp://academiccommons.columbia.edu/catalog/ac:110742
Murphy, Christian; Kim, Eunhee; Kaiser, Gail E.; Cannon, Adamhttp://hdl.handle.net/10022/AC:P:29518Wed, 27 Apr 2011 00:00:00 +0000The errors that Java programmers are likely to encounter can roughly be categorized into three groups: compile-time (semantic and syntactic), logical, and runtime (exceptions). While much work has focused on the first two, there are very few tools that exist for interpreting the sometimes cryptic messages that result from runtime errors. Novice programmers in particular have difficulty dealing with uncaught exceptions in their code and the resulting stack traces, which are by no means easy to understand. We present Backstop, a tool for debugging runtime errors in Java applications. This tool provides more user-friendly error messages when an uncaught exception occurs, but also provides debugging support by allowing users to watch the execution of the program and the changes to the values of variables. We also present the results of two studies conducted on introductory-level programmers using the two different features of the tool.Computer scienceek2044, gek1Computer ScienceTechnical reportsDistributed In Vivo Testing of Software Applicationshttp://academiccommons.columbia.edu/catalog/ac:110800
Chu, Matt; Murphy, Christian; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:29536Wed, 27 Apr 2011 00:00:00 +0000The in vivo software testing approach focuses on testing live applications by executing unit tests throughout the lifecycle, including after deployment. The motivation is that the "known state" approach of traditional unit testing is unrealistic; deployed applications rarely operate under such conditions, and it may be more informative to perform the testing in live environments. One of the limitations of this approach is the high performance cost it incurs, as the unit tests are executed in parallel with the application. Here we present distributed in vivo testing, which focuses on easing the burden by sharing the load across multiple instances of the application of interest. That is, we elevate the scope of in vivo testing from a single instance to a community of instances, all participating in the testing process. Our approach is different from prior work in that we are actively testing during execution, as opposed to passively monitoring the application or conducting tests in the user environment prior to execution. We discuss new extensions to the existing in vivo testing framework (called Invite) and present empirical results that show the performance overhead improves linearly with the number of clients.Computer sciencemwc2110, gek1Computer ScienceTechnical reportsA Distance Learning Approach to Teaching eXtreme Programminghttp://academiccommons.columbia.edu/catalog/ac:110823
Murphy, Christian; Phung, Dan; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:29543Wed, 27 Apr 2011 00:00:00 +0000As university-level distance learning programs become more and more popular, and software engineering courses incorporate eXtreme Programming (XP) into their curricula, certain challenges arise when teaching XP to students who are not physically co-located. In this paper, we present the results of a three-year study of such an online software engineering course targeted to graduate students, and describe some of the specific challenges faced, such as students' aversion to aspects of XP and difficulties in scheduling. We discuss our findings in terms of the course's educational objectives, and present suggestions to other educators who may face similar situations.Computer sciencegek1Computer ScienceTechnical reportsA Uniform Programming Abstraction for Effecting Autonomic Adaptations onto Software Systemshttp://academiccommons.columbia.edu/catalog/ac:110713
Valetto, Giuseppe; Kaiser, Gail E.; Phung, Danhttp://hdl.handle.net/10022/AC:P:29509Wed, 27 Apr 2011 00:00:00 +0000Most general-purpose work towards autonomic or self-managing systems has emphasized the front end of the feedback control loop, with some also concerned with controlling the back end enactment of runtime adaptations -- but usually employing an effector technology peculiar to one type of target system. While completely generic "one size fits all" effector technologies seem implausible, we propose a general purpose programming model and interaction layer that abstracts away from the peculiarities of target specific effectors,enabling a uniform approach to controlling and coordinating the low-level execution of reconfigurations, repairs,micro-reboots, etc.Computer sciencegek1Computer ScienceTechnical reportsSelf-Managing Systems: A Control Theory Foundationhttp://academiccommons.columbia.edu/catalog/ac:110718
Diao, Yixin; Hellerstein, Joseph L.; Parekh, Sujay; Griffith, Rean; Kaiser, Gail E.; Phung, Danhttp://hdl.handle.net/10022/AC:P:29510Wed, 27 Apr 2011 00:00:00 +0000The high cost of ownership of computing systems has resulted in a number of industry initiatives to reduce the burden of operations and management. Examples include IBM's Autonomic Computing, HP's Adaptive Infrastructure, and Microsoft's Dynamic Systems Initiative. All of these efforts seek to reduce operations costs by increased automation, ideally to have systems be self-managing without any human intervention (since operator error has been identified as a major source of system failures). While the concept of automated operations has existed for two decades, as a way to adapt to changing workloads, failures and (more recently) attacks, the scope of automation remains limited. We believe this is in part due to the absence of a fundamental understanding of how automated actions affect system behavior, especially system stability. Other disciplines such as mechanical, electrical, and aeronautical engineering make use of control theory to design feedback systems. This paper uses control theory as a way to identify a number of requirements for and challenges in building self-managing systems, either from new components or layering on top of existing components.Computer sciencerg2023, gek1Computer ScienceTechnical reportsThe In Vivo Approach to Testing Software Applicationshttp://academiccommons.columbia.edu/catalog/ac:110832
Murphy, Christian; Kaiser, Gail E.; Chu, Matthttp://hdl.handle.net/10022/AC:P:29546Wed, 27 Apr 2011 00:00:00 +0000Software products released into the field typically have some number of residual bugs that either were not detected or could not have been detected during testing. This may be the result of flaws in the test cases themselves, assumptions made during the creation of test cases, or the infeasibility of testing the sheer number of possible configurations for a complex system. Testing approaches such as perpetual testing or continuous testing seek to continue to test these applications even after deployment, in hopes of finding any remaining flaws. In this paper, we present our initial work towards a testing methodology we call in vivo testing, in which unit tests are continuously executed inside a running application in the deployment environment. These tests execute within the current state of the program (rather than by creating a clean slate) without affecting or altering that state. Our approach can reveal defects both in the applications of interest and in the unit tests themselves. It can also be used for detecting concurrency or robustness issues that may not have appeared in a testing lab. Here we describe the approach and the testing framework called Invite that we have developed for Java applications. We also enumerate the classes of bugs our approach can discover, and provide the results of a case study on a publicly-available application, as well as the results of experiments to measure the added overhead.Computer sciencegek1, mwc2110Computer ScienceTechnical reportsProperties of Machine Learning Applications for Use in Metamorphic Testinghttp://academiccommons.columbia.edu/catalog/ac:110846
Murphy, Christian; Kaiser, Gail E.; Hu, Lifenghttp://hdl.handle.net/10022/AC:P:29550Wed, 27 Apr 2011 00:00:00 +0000It is challenging to test machine learning (ML) applications, which are intended to learn properties of data sets where the correct answers are not already known. In the absence of a test oracle, one approach to testing these applications is to use metamorphic testing, in which properties of the application are exploited to define transformation functions on the input, such that the new output will be unchanged or can easily be predicted based on the original output; if the output is not as expected, then a defect must exist in the application. Here, we seek to enumerate and classify the metamorphic properties of some machine learning algorithms, and demonstrate how these can be applied to reveal defects in the applications of interest. In addition to the results of our testing, we present a set of properties that can be used to define these metamorphic relationships so that metamorphic testing can be used as a general approach to testing machine learning applications.Computer sciencegek1, lh2342Computer ScienceTechnical reportsA Framework for Quality Assurance of Machine Learning Applicationshttp://academiccommons.columbia.edu/catalog/ac:110590
Murphy, Christian; Kaiser, Gail E.; Arias, Martahttp://hdl.handle.net/10022/AC:P:29471Wed, 27 Apr 2011 00:00:00 +0000Some machine learning applications are intended to learn properties of data sets where the correct answers are not already known to human users. It is challenging to test and debug such ML software, because there is no reliable test oracle. We describe a framework and collection of tools aimed to assist with this problem. We present our findings from using the testing framework with three implementations of an ML ranking algorithm (all of which had bugs).Computer sciencegek1Computer Science, Center for Computational Learning SystemsTechnical reportsExperiences in Teaching eXtreme Programming in a Distance Learning Programhttp://academiccommons.columbia.edu/catalog/ac:110774
Murphy, Christian; Phung, Dan; Kaiser, Gail E.http://hdl.handle.net/10022/AC:P:29528Wed, 27 Apr 2011 00:00:00 +0000As university-level distance learning programs become more and more popular, and software engineering courses incorporate eXtreme Programming (XP) into their curricula, certain challenges arise when teaching XP to students who are not physically co-located. In this paper, we present our experiences and observations from managing such an online software engineering course, and describe some of the specific challenges we faced, such as students' aversion to using XP and difficulties in scheduling. We also present some suggestions to other educators who may face similar situations.Computer sciencegek1Computer ScienceTechnical reports