You need to understand how the packet goes and whether this path is correct from your point of view. This applies to both, incoming and outgoing traffic.

Packet paths can be observed via the tcpdump tool. In most of the cases you should launch the tcpdump -i <if> -e host <ip>, where the <if> is the interface name you're experience problems with and the <ip> is the IP address relevant to you traffic (e.g. if you can't ping kernel.org, then the <ip> is the kernel.org's IP).

For a generic case of venet VE outgoing packet should flow via VE's venet0 interface, VE0 venet0 interface and ethN in the HN.

So, you should observe the packet on each interface above. Incoming packets fly in the reversed order and you also should observe them there.

In case veth device is used packets also pass bridge interfaces in VE0.

With this information provided, the problem can at least be tried to be solved.