@PinkyK 

It sounds like you're trying to offset the amount you owe them (A/P) with the amount they owe you (A/R) or vice versa.  Am I understanding that correctly?  If so, credit memos are not the best way to handle it.  The easiest way IMO is to create a bank account in QB called "Clearing Account".  Then, pay their vendor bill using the Clearing Account.  Then, Receive Payment on the invoice in the same amount and deposit it into the Clearing Account.  You now have their vendor bill paid, the Clearing Account is $0, and their customer invoice has been paid down by the amount of the bill.  The amount due on their invoice should be invoice total less their vendor bill. 

If their vendor bill is more than their customer invoice, just reverse the process - receive payment on the invoice in full and deposit it into the Clearing Account.  Then pay down their vendor bill by the same amount from the Clearing Account.  Now, the balance due on the vendor bill is the total bill amount less their customer invoice.