i finally installed the NSM
my mistake was i didnt reboot the server after i confired the shared memory (by putting the kernel.shmmax= 402653184 in /etc/sysctl.conf)and installing the system updates.
i just installed the NSM right away after doing that.
what i did is resintalling the RHEL, and then doing the installation and configure the shared memory, reboot and after rebooting installed the NSM, and it was installed successfully.