One of the concerns sellers have against seller financing is a buyer able to refinance and paying back a loan soon after closing the sale. You can make sure there’s a prepayment penalty in the note to prevent most buyers to pay early. There are limitations to what the note holder can charge and for how long.
Every situation involves risk and the risk risk of the borrower paying the note quickly, is not a risk but a tax liability, so the worse situation is paying taxes on the payoff but the longer the buyer pays, should offset a lot of that tax burden. If that happens consult your CPA or tax expert.