Well I've owned the 100-400, 70-200 f2.8, both IS and non IS, two 70-200 f4 L's and the 70-200 f4 L IS which is currently the only one of the above I still have.
The 100-400, especially when zoomed, feels a little heavier/bulkier then the f2.8 IS but similar when not zoomed. Since the f2.8 IS has internal zoom that's not really an issue with the 70-200's.
Everyone will tell you I've been pretty overzealous in my praise of the 70-200 f4 L IS because of the it's image quality, it's simply the sharpest lens I own. The shots just pop off the page.
There's a popular axiom that if you have to ask if you need f2.8 you probably don't but of course that doesn't mean you won't find out later that you do!
The 70-200 f4 L non IS is probably the biggest bargain out there lens wise and worth considering as well. The two I owned were OK but terrible with the 1.4 convertor on. This was a while back though and many of the original models suffered from front focus problems which very well could have been the case with the two I owned.
The sports pros love the f2.8L IS because of it's bokeh and they'll just switch to 300 or 400's when they need more length. One of the things I love about the f4L IS is that I can throw my 1.4 TC on it and still have a super sharp hand holdable 280mm that's still a reasonably compact package.
There is no perfect answer here you just need to think about what you're going to be shooting and what best fits that.