Is it fair to the person who is potentially libeled to be forced to travel to the state of the speaker to file suit??

The most harm will occur where the person allegedly libeled lives -- so the state where he lives is probably the best location for a suit. If I live in Detroit and some knob in Alabama says I am a white supremacist, the damages to me are in Detroit, where I lose business, not Alabama where the accuser lives. Detroit is the best location for the suit.

Libel is a very high bar to clear -- so actual filed libel suits are rare birds indeed. Nationwide, they might hit the hundreds each year. That's with 300,000,000 people.