This paper describes the Building Controls Virtual Test Bed (BCVTB) that is currently under development at Lawrence Berkeley National Laboratory. An earlier prototype linked EnergyPlus with controls hardware through embedded SPARK models and demonstrated its value in more cost-effective envelope design and improved controls sequences for the San Francisco Federal Building. The BCVTB presented here is a more modular design based on a middleware that we built using Ptolemy II, a modular software environment for design and analysis of heterogeneous systems. Ptolemy II provides a graphical model building environment, synchronizes the exchanged data and visualizes the system evolution during run-time. Our additions to Ptolemy II allow users to couple to Ptolemy II a prototype version of EnergyPlus, MATLAB/Simulink or other simulation programs for data exchange during run-time. In future work we will also implement a BACnet interface that allows coupling BACnet compliant building automation systems to Ptolemy II. We will present the architecture of the BCVTB and explain how users can add their own simulation programs to the BCVTB. We will then present an example application in which the building envelope and the HVAC system was simulated in EnergyPlus, the supervisory control logic was simulated in MATLAB/Simulink and Ptolemy II was used to exchange data during run-time and to provide real-time visualization as the simulation progresses.