1A "capitalist" is not a supporter of capitalism but a person with a lot of capital. In my own rough third millennium terms, if you own less than one hundred million, net, you are not a capitalist.
2 Ronald Coase ("The Lighthouse in Economics") points out that the tax might be tied to use of a nearby port, and claims that would be a free-market solution. I doubt that any Republican would be fooled by that -- a tax is a tax is a tax, unless somebody else is paying it.
3 Three qualifications are necessary here that are digressive, and so belong in a footnote, but that are necessary to get the point quite right. First, some public goods are provided in expectation of profits. For example, entertainment broadcasts are public goods. But broadcast advertising is a private good, and broadcasters are able to bundle the public and private good together. This may not be efficient, beautiful or suitable for a family audience, but it does provide the public good. Second, many of the goods that are neglected probably would not exactly fulfill the conditions as the lighthouse does. Strictly speaking, they would not be public but quasipublic goods: goods whose costs structures are largely, but not 100%, overheads, and for which it is not impossible, but merely inconvenient, to make payment a condition of benefit. Finally, there may be other reasons beside quasipublic goods status that would make it economically efficient to supply goods outside the price system. But public goods are the best understood of these cases in modern economics.
4 In case it is not obvious, though I think it should be, that goes double for anarchist capitalism.
5 James H. Rubin, "Business profits in 1990s fueled by stagnant wages, report says," The Philadelphia Inquirer, Sunday, Sept. 3, 1995, p. A4.