Configuration variable "mapred.child.ulimit" can be used to control the maximum virtual memory of the child (map/reduce) processes.

** value of mapred.child.ulimit > value of mapred.child.java.opts