2017-11-13 18:46 GMT+08:00 Peter Zijlstra <peterz@infradead.org>:> On Mon, Nov 13, 2017 at 04:26:57PM +0800, Wanpeng Li wrote:>> 2017-11-13 16:04 GMT+08:00 Peter Zijlstra <peterz@infradead.org>:>>> > So if at this point a vCPU gets preempted we'll still spin-wait for it,>> > which is sub-optimal.>> >>> > I think we can come up with something to get around that 'problem' if>> > indeed it is a problem. But we can easily do that as follow up patches.>> > Just let me know if you think its worth spending more time on.>>>> You can post your idea, it is always smart. :) Then we can evaluate>> the complexity and gains.>> I'm not sure I have a fully baked idea just yet, but the general idea> would be something like:>> - switch (back) to a dedicated TLB invalidate IPI>> - introduce KVM_VCPU_IPI_PENDING>> - change flush_tlb_others() into something like:>> for_each_cpu(cpu, flushmask) {> src = &per_cpu(steal_time, cpu);> state = READ_ONCE(src->preempted);> do {> if (state & KVM_VCPU_PREEMPTED) {> if (try_cmpxchg(&src->preempted, &state,> state | KVM_VCPU_SHOULD_FLUSH)) {> __cpumask_clear_cpu(cpu, flushmask);> break;> }> }> } while (!try_cmpxchg(&src->preempted, &state,> state | KVM_VCPU_IPI_PENDING));> }>> apic->send_IPI_mask(flushmask, CALL_TLB_VECTOR);>> for_each_cpu(cpu, flushmask) {> src = &per_cpu(steal_time, cpu);> smp_cond_load_acquire(&src->preempted, !(VAL & KVM_VCPU_IPI_PENDING);> }>>> - have the TLB invalidate handler do something like:>> state = READ_ONCE(src->preempted);> if (!(state & KVM_VCPU_IPI_PENDING))> return;>> local_flush_tlb();>> do {> } while (!try_cmpxchg(&src->preempted, &state,> state & ~KVM_VCPU_IPI_PENDING));

There are a lot of cases handled by flush_tlb_func_remote() ->flush_tlb_function_common(), so I'm afraid to have hole.

Regards,Wanpeng Li

>> - then at VMEXIT time do something like:>> state = READ_ONCE(src->preempted);> do {> if (!(state & KVM_VCPU_IPI_PENDING))> break;> } while (!try_cmpxchg(&src->preempted, state,> (state & ~KVM_VCPU_IPI_PENDING) |> KVM_VCPU_SHOULD_FLUSH));>> and clear any possible pending TLB_VECTOR in the guest state to avoid> raising that IPI spuriously on enter again.>>> This way the preemption will clear the IPI_PENDING and the> flush_others() wait loop will terminate.