How can I turn off or limit file history tracking?

File manager and video player keep tracking my recently used files and folders, but I wish to keep my file usage private.