Structural condition assessment of highway bridges is traditionally performed by visual inspections or nondestructive evaluation techniques, which are either slow, unreliable or detects only local flaws. Instrumentation of bridges with accelerometers and other sensors, however, can provide real-time data useful for monitoring the global structural conditions of the bridges due to ambient and forced excitations. This paper reports a video-assisted approach for structural health monitoring of highway bridges, with results from field tests and subsequent offline parameter identification. The field tests were performed on a short-span instrumented bridge. Videos of vehicles passing by were captured, synchronized with data recordings from the accelerometers. For short-span highway bridges, vibration is predominantly due to traffic excitation. A stochastic model of traffic excitation on bridges is developed assuming that vehicles traversing a bridge (modeled as an elastic beam) form a sequence of Poisson process moving loads and that the contact force of a vehicle on the bridge deck can be converted to equivalent dynamic loads at the nodes of the beam elements. Basic information of vehicle types, arrival times and speeds are extracted from video images to develop a physics-based simulation model of the traffic excitation. This modeling approach aims at circumventing a difficulty in the system identification of bridge structural parameters. Current practice of system identification of bridge parameters is often based on the measured response (or system output) only, and knowledge of the input (traffic excitation) is either unknown or assumed, making it difficult to obtain an accurate assessment of the state of the bridge structures. Our model reveals that traffic excitation on bridges is spatially correlated, an important feature that is usually incorrectly ignored in most output-only methods. A recursive Bayesian filtering is formulated to monitor the evolution of the state of the bridge. The effectiveness and viability of this video-assisted approach are demonstrated by the field results.