This is the same problem I have been having now for some time with Quickbooks online. 

You can not Apply a Credit Memo to an Invoice w/out receiving payment from customer.

If you go to Receive Payment to ONLY apply a Credit Memo it doesn't work. 

Amount Received zero, check the invoice with and enter amount of credit in Payment box. Then check the Credit Memo for same amount you get a negative amount in the Apply Credit section at end. Save and Close and you get a warning message that a negative Credit can not be applied.

However, I did find that if you go the the Customer click the green New Transaction drop down and select Payment it automatically applies the Credit Memo to the Invoice w/ a zero Amount Received and Apply Amount at the end is the Credit amount. Basically this is the only way I have found it to work properly.

The other features such as going through the Received Payment button or drop down arrow next to invoice on the customer info page will not work in the correct way to apply payment the way it should.

Major flaw with QB online with applying Credit Memos. 

Hopefully they can fix this and avoid a lot of clients headaches.