This came up the other day when I was talking with Brenda Laca over at the UAB. Consider:
We had lunch at twelve.
I agree (tentatively) with Prof. Laca that this means, as a matter of semantics, that the actual eating started at twelve. Thus it would be wrong to assert this sentence if the lunch went from eleven to one; in that case the right thing to say would be
We were having lunch at twelve.
Still, it seems to me that even in these circumstances, it wouldn't be right to contest the first assertion, at least not by saying something like
No you didn't!
If anyone shares these intuitions or has different ones, I'd be grateful to read about them!