In recent years, cloud computing has become one of the most heated terms in the information industry. Meanwhile, as a key supportive technique of cloud computing, virtualization technology has achieved incredible development. However, legacy security issues still exist in virtual cloud computing environment and become more harmful, and the wide adoption of cloud computing renders these problems increasingly serious. Among them, software security has always been a central topic that underlies many other security problems and deserves great attention. Software security can usually be divided into two complementary aspects: the first is to prevent software execution from causing damage to operating systems or other applications, while the second is to protect the execution of software against attacks from the external environment. Conventionally, sandbox mechanisms can be used to resolve software security issues. However, they suffer from several notable deficiencies under the virtualized environment: firstly, the definition of conventional sandbox only pays attention to the first aspect of software security, and ignores the second aspect; secondly, existing sandbox mechanisms could be bypassed or even exploited by attackers due to some flaws in their design and implementation; finally, it is inherently difficult to deploy and adapt sandboxes that run inside the operating systems for virtual cloud computing environment. Therefore, it is an urgent task to resolve software security issues under virtualized environment.

Therefore, it is an urgent task to resolve software security issues under virtualized environment. This dissertation targets the two aspects of software security that are mentioned above: the investigation in the first aspect integrates conventional sandbox mechanisms in virtual machine monitor to provide better robustness and ease-of-adoption, and the research in the second aspect provides a solution to return oriented programming, the most threatening attack to software nowadays. The main contribution and novelty of this dissertation include:

1. Extension of the sandbox concept to include software execution protection. Traditional sandbox mechanisms only prevent software execution from harming the system. However, as an execution environment, sandboxes could and should also protect software execution against external attacks. These two aspects are complementary. Namely, some applications behave maliciously (as attackers) just because they have been attacked by external adversaries (as victims). A typical example is that an attacker exploits the higher privilege of a program that unfortunately has a buffer overflow vulnerability for malware execution.

2. A novel algorithm to defend return oriented programming. Harnessing an essential feature of return oriented programming, i.e., putting a host of library return addresses on stack and triggering the attack with ret instruction, this dissertation proposes a novel countermeasure for this attack. By means of analyzing the ratio of stack elements within the library range before each ret instruction is executed and comparing the ratio to a specified threshold, it is feasible to recognize a potential attack and make alarms beforehand. This approach is suitable for virtualized environment and has small performance overhead and low false positive/negative rate.

3. An application sandbox framework based on virtualized environment. By deploying sandbox mechanisms inside the virtual machine monitor which has the highest privilege, the framework prevent attacks such as "time of check to time of use" race conditions. This dissertation first uses a design pattern that dynamically load executable code to hypervisors at runtime, which avoids performance degradation caused by frequent context switches, and creatively utilizes a simplified programming language for policy description to provide high flexibility.

4. Design, implementation and evaluation of two proof-of-concept systems. Based on the open source Xen virtual machine monitor, this dissertation has designed and implemented two prototype systems for the above work, HyperCrop and VCCBox respectively, and has performed effectiveness and efficiency experiments on them. The results of evaluation demonstrate that both systems can reach their protection goals and only introduce accepted performance overhead, rendering their usefulness in real production workload.