"if I want to keep a the amount returned as a balance in the petty cash box and not deposit it to the checking account, how would I enter that?"

You already have it.

Like I stated:

You take from Checking = it appears in the Petty Cash Bank, representing that money in that person's possession right now.

They bring you the receipts from Spending and their Change.

You enter the Spending = the Petty Cash bank account has a Left Over Balance = not everything was spent. What you are Holding is that amount. You are Holding Cash On Hand, which is being tracked as Petty Cash.

In the Petty Cash bank in QB, you see the balance that you got handed and put into the Box.