Ability to configure relative-to fields

I would like to configure the 'Relative to' field so that I can select the custom date fields that appear on that dropdown. Right now I cannot restrict the custom fields that appear on the drop-down without removing that field from that project or issue type. I would like to keep the field on that project but just hide it from the dropdown.