How can I enter previous layaways before RMS was installed, record a deposit, and collect the correct amount due?

Option A: Assuming you have NOT collected sales tax or declared the product as sold, you can create transaction representing the original sale.
Create a Non-inventory item code called PRIOR LAYAWAY with the retail price set to $0.00 and prompt for price at POS checked off (Options tab).
Sell this item to the customer account at the original retail price and record a "fake" cash deposit. Add a slip to the drawer representing this cash amount.
When the customer comes in they owe the balance and you can view the deposit amount previously paid.

Option B: Assuming you HAVE collected sales tax and declared the product as sold previously, you can add account status to the customer account and make a manual adjustment to that account reflecting the balance due (debit).
When the customer comes in they will owe that amount.