If you want a breakdown as to why, consider this. What function does "had" have in the phrase? Is it being considered an auxiliary verb? If so, why isn't "do" conjugated as "done"? Is it being used in the sense of an obligation? If so, why do we have "do" instead of "to do"? In both cases, why do we have "had", a past tense conjugation, for present expectations?
I think splitting hairs on proper grammar for a phrase that isn't "proper" grammar to begin with is rather meaningless. Its meaning is understood regardless.