Yes, that bracket shouldn't move, and if the seat lines up and bolts go through their holes correctly, then it hasn't moved.
I know mine has been in a bind before and not seated fully or correctly, causing the latch to not fully engage the hole at the upper bracket... usually a good whack with the palm of my hand will seat / latch it. My guess is that it is not going flush enough with the bracket up top, which could be caused by the lower bracket not allowing movement (it pivots from vertical to leaning out so you can remove the tool box). Try a bit of WD-40 on the lower bracket to free it up, or see if anything is preventing it from moving freely.
2002 VT1100C2 Sabre
1985 VT700C Shadow
1999 CB750 NightHawk (son's ride)
"The Hardest Part About a Zombie Apocalypse, Will Be Pretending I'm Not Excited