There have been threads over on the official forums discussing this and potential solutions (SSDs, RAID 0 setups, Gigabyte iRAM, etc). It really ought to keep more things in RAM than it does. Basically, I think it's I/O limited in many ways. It also starts new scripts for things as you enter different areas.

There are problems with how the game streams data in as you move around the world. Click to expand.I'm not convinced that the GPU is the cause of Oblivion's stutters.