ZFS is very memory intensive by default, since it uses you RAM as a cache. Because of that is is also very fast. If you don’t want zfs to use that much memory and are willing to accept the performance hit, you can set limits on the memory usage of zfs.

This Procedure is described in the Proxmox Documentation for ZFS.

Check how much memory is used

First you should check how much memory zfs is using at the moment. Maybe ZFS is not your memory bottleneck after all.

You can do so by reading a file in the procfs.

[tux@server]$ cat /proc/spl/kstat/zfs/arcstats

In there you will find the lines c, c_max and size.
c is the target size of the ARC cache in Bytes.
c_max is the maximum size of the ARC cache in Bytes.
size is the current used memory for the ARC cache.

Limit the Memory usage

You can set the minimum and maximum amount of memory used by zfs by creating a zfs.conf in the /etc/modprobe.d/ directory.

[tux@server]$ vim /etc/modprobe.d/zfs.conf

# Don't let ZFS use less than 4GB and more than 64GB
# The numbers are in Bytes
options zfs zfs_arc_min=4294967296
options zfs zfs_arc_max=68719476736

Change the numbers to what every is appropriate for your machine and use case.

You need to reboot your system for these changes to take effect.