As a first step, what I would do is look at the processes running to see if anything in particular stands out. You can use the "show system processes" command for that, or jump to a shell and run the "top" command. I prefer top as it lets me sort the output easily by memory usage (hit the "m" key twice to have the highest memory usage processes on top of the list). Based on which process(es) are taking excessive memory, the next steps would vary greatly.
Ron