Note that if you let your domain lapse and someone else registers it afterwards, that person will also gain control over your e-mail address (and likely all accounts associated with it, if they are not secured with an additional factor of authentication / recovery).
As long as you’re very thorough about removing any linked connection from your expired e-mail, you should be OK. That includes all accounts that you registered using this e-mail, as well as all e-mail contacts that you’ve built up using that account.