Etymologies

brick + bat1, piece of brick.

(American Heritage® Dictionary of the English Language, Fourth Edition)

brick +‎ bat Bricklaying jargon for broken bricks that cannot be used is 'bats'. The connotation is that they are useless for building purposes. Hence a brick bat is an item of negligible worth. (Wiktionary)

Examples

I do not feel any dereliction of duty but do feel that your brickbat is a duffer.

As described spot-on by Timothy Garton Ash in The Guardian newspaper, "In France, genocide has become a political brickbat," Jan. 18: ... a tragedy which should be the subject for grave commemoration and free historical debate, calmly testing even wayward hypotheses against the evidence, is reduced to an instrument of political manipulation, a politician's brickbat.