A tense thriller and a family-friendly sequel bumped "Guardians of the Galaxy" from the top of the weekend's box office.