I figured it out! You have to whitelist your domain for the chat box to show up. This is not in the area where you set up the chat box code but above it.
Go to Messenger Platform Settings on your page Scroll down to: Whitelisted Domains Third-party domains that are accessible in the Messenger webview for use with the Messenger Extensions SDK, and for Messenger plugins.
Then set up your Customer Chat Plugin and inject the code in the footer under advanced settings.
@atticusmabry @figlet