Are you sure libfm depends on gtk-doc?
It looks like it should be optional:
configure options:
--enable-gtk-doc use gtk-doc to build documentation [[default=no]]
--enable-gtk-doc-html build documentation in html format [[default=yes]]
--enable-gtk-doc-pdf build documentation in pdf format [[default=no]]
From configure output:
Build API doc with gtk-doc (required for make dist): no