I seem to lose all my shards when I move my soulshard bag to my bank (I do this when I need my enchanting mats from the bank, just switch the 2 bags).
Have also lost all my shards at a login once but have no idea what caused it.