I didn’t remember this being part of the rules, so I looked up Milton-Bradley’s official rules, and you know what I found?
It’s not part of the official rules anyway. The rules don’t specify anything about where you can pull blocks from, or how, beyond: Blocks must be removed from below the highest completed level of the tower, and you may only use one hand to remove a block… Neither of those rules was violated here.
As long as the tower hasn’t fallen, the game is still going.