Kind of hard to imagine, but there was a time. The BBC posted an article that discusses the history and issues.

Do passports restrict economic growth?