Dang... you have to have a clip missing or something... are you SURE all the Parts are THERE ?!?!
You are just going to have to track down the EXACT source of the Problem. Even if it means driving without a door panel till it happens.. and then go get the part in a junk yard ..