All righty then...
I'm assuming you've got the full version of FS2, patched to 1.2. If you have the Home of the Underdogs version (which, because of legal issues, I won't link to; use Google if you must), you'll need to download some extra files, which are available from a community FTP server. You can find details about it in
this thread and details about which files you'll need on
this page. At any rate, after you have a full version of FS2 patched to 1.2, you need to download the FS2_Open files. The first thing you'll need is the actual executable file. The most recent official release, 3.6.5, is available in the thread that MD linked to. However, I'd highly recommend this more recent test build. Among other things, it now featuers specular mapping for OpenGL.
Here it is. Keep an eye on the recent builds forum that MD linked to; most of the newer builds in there contain newer features than the last official release. The next file you'll need is the new launcher; you can find that
here. If you're interested in creating FS2 missions, you'll need the latest copy of FRED, available
here. Up to now, installation is easy; just put the launcher, executable (with included .dll files), and FRED in your main Freespace 2 directory. They won't overwrite anything. I'd also recommend making a shortcut to the launcher on the desktop; it makes it easier to start everything up.
Now you'll need to download the VP files. These contain all of the new models, textures, sounds, and effects used by FS2_Open. You can find all of the VP files on
this page. Note that "mv_zpack" contains all of the VP files except for "mv_aseffects." Also, unless you have a very slow computer, you shouldn't need "mv_loplanets." Once you download all of these files, you'll need to put them in the FS2 directories. You can just dump them in the main folder, but I prefer something different. The launcher supports a "-mod" flag that specifies a certain folder where all the files for a particular mod are kept. This makes it very easy to change mods by just selecting a differnt folder. That being said, I'd advise creating a folder called "SCP" or "Enhanced" in your main FS2 folder and installing the VP files in there.
Now, you're ready to run the launcher. One side note: to configure a joystick, you'll have to use the original FS2 launcher, since the Open launcher doesn't have a tab for it. On the first page of the launcher, you'll see a "Browse" button in the upper-right, use it to find your FS2_Open.exe. Next, go to the Video tab and select Direct3D or OpenGL. You might have to experiment to see which runs better for you, but if you have an ATI card, you'll need to use OpenGL to see specular mapping. Unfortunately, OpenGL doesn't support environmental mapping right now, but the shinemaps are more than worth it. After this, go to the Features tab. In this tab, go to "Select Mod" and choose the folder where you installed the VP files. (Important note: folder names must be one word only.) Next, scroll down the different list types and start turning on features. Here's what I have on; this represents just about everything you need: specular, glowmaps, 32-bit textures (although this can cause performance issues), mipmapping, stop fog intersect hull (use only in D3D), another pair of scanning lines (just aesthetic), target info next to target (very useful), 3D warp (by preference; you should also check out 3D radar if you're interested), environmental mapping, and use alpha for env mapping. After this, hit "Apply" to save your settings, and you're all set!
Plus, our very own Inquisitor is the project manager. 
So I downloaded one called "WAR" and put it in its own folder like MD has with the TBP folder. It shows up in Select Mod but when I highlite it and hit O.K. I get the same "Mod directory name can't contain space" message.
