slabtop command (part of the package procps) shows top memory objects used by the kernel.
dstat can help you figure out what is happening. dstat -cdnpmgs --top-bio --top-cpu --top-mem
Also have a look at smem ("smem -kt"), it can show you nicely what is in your swap.