This is an area that I know a fair bit about, as I work in environmental and fisheries consulting.First off, the environmental effects and the impacts of fishing entirely depends what fish, what population (i.e., what lot of fish breed where), and how they're fished. Some fish are still plentiful, and making lots more of themselves. Some aren't. You can't make blanket comments like "I shouldn't eat so much fish," as that's essentially meaningless.
Fish farms are a different kettle of, well, fish from wild caught fish, and they've had some very unpleasant impacts. I'm not willing to condemn them all, though -- there's no reason why fish farming can't be done right. But how it's been managed so far in many areas has been hugely destructive, ranging from the demolition of huge swaths of mangroves for shrimp production in south Asia, the complete industrialization of large parts of the coast of China for mind-bogglingly intensive aquaculture, and the messes resulting from salmon farms on the coasts of Canada.
But many fisheries are sustainable, and should be supported and encouraged. I regularly buy hook-and-line caught haddock from my local organic food supplier. I buy lobster, clams, farmed mussels. I buy mackerel and herring. Fish is a *good* thing.
You can educate yourselves about what's working and what's not in fisheries fairly easily -- take a look at Seaweb, for example, http://www.seaweb.org/try, and their highly successful "Give Swordfish a Break" campaign. Or google "Take a Pass on Chilean Sea Bass". Or check out the Marine Stewarship Council, http://www.msc.org/, or Nova Scotia's Ecology Action Centre, http://www.ecologyaction.ca/marine_issues/marine_issues.shtm, and their ongoing fight to stop dragging/trawling.
The increasing pressure to avoid pressure on the vulnerable species and populations, and to encourage the use of the sustainable ones, is working. Many fish populations are critically endangered, no question -- but I would argue that people are becoming increasingly aware of the implications, and protection campaigns are working. Don't stop eating fish -- just eat smart.