Hi @dylanthered.
Unfortunately, there is no option to remove a linked record with a one-click action button. You can either clear all values or replace them.
If you have a button to add an item to your favorites and want to have another button to remove it from favorites, here is an alternative workaround I could suggest.
You need to have two linked records in your base: one for the Add to favourites button and one for Remove from favourites buttons. Please check the screenshot below for reference.
Then you need to put a conditional filter that meets both the conditions. This way, the user will see the records in their Favourite page as long as they does not mark it as Not Favourite. Please check the screenshot below for reference for conditional filters.
The example is based on Add to Favourites and Remove from Favourites buttons, but the wording is flexible and you can use it for your use case as needed.