Its OK to me but I am not crazy about it. Why keep trying it if you don't like it though?
I don't like peanut butter. I do love hazelnut butter and pistachio butter though. There are far too many yummy things you can have eating this way to eat things you don't like. If you don't like almond butter, I don't think not liking the almond butter bread was a fluke.

I hate any "real" nut butter that is made with only ground nuts... esp peanut butter. It tastes chalky and oily at the same time to me. Unfortunately, I like the processed kind with oil and sugar added (think Jif or Skippy)... therefore I basically only rarely buy any.

I did find that it works as a tahini substitute when making hummus or baba ganoush.