I've received a few requests via private messages for how to fix this, so I'm going to go ahead and post my solution here.
First, I will make the disclaimer that this is not supported by Juniper. This fix is my own work, it was not suggested by JTAC nor is running NSM on a 64-bit platform endorsed or supported by JTAC.
That being said, it's really quite an easy fix and I have not had any problems running on RHEL 5.x x86_64 for over 3 years. The fix is to basically lie to the installer script. At a bash command prompt on any linux/unix workstation, this should work. The source file is the NSM installer script for the version you're installing. I'm going to use nsm2010.4 as an example.
[keithr@nsm ~]$ sed 's/ARCH=`\/bin\/uname -m`/ARCH="i686"/' nsm2010.4_servers_linux_x86.sh > my-nsm.sh
Note that the above command is all one line, in case the format is affected by the forum posting.
Simply then install NSM from the new "my-nsm.sh" script.
I also keep my system updated via RHN and yum, which again is not supported by Juniper. I feel, though, that keeping my system patched with security-related updates from RHN is essential for a server that has master control of all my firewall devices. I do not use the systemupdate scripts, instead I extract the systemupdate zip and tar files, then go into the es5 directory that is created. I find the rhes5.sh script to be naive and clunky, since all it really does is this:
rpm -Uvh --force --nodeps *.rpm
That can be damaging to your system if you have run any security updates or patches on it.
For a fresh install, I catalog the RPMs that are in the systemupdate (es5) directory and make sure that the same or newer versions are installed on my system using yum. Some of the supplied packages in the systemupdate tar file won't install because they aren't targeted for the x86_64 platform, so you may have to do a little homework and find the correct packages and install them with yum.
Your mileage may vary, don't try this at home, professional driver - closed course... etc.