Later than most people assume, and the answer depends on your pathway.
For a 510(k), the quality system is not required until you sell the device. You can submit without it. Teams routinely stand up the QMS after the submission goes in, while FDA is reviewing.
For a De Novo, it should be running during the review. Scrutiny is higher, post-market expectations are heavier, and FDA can audit during the review. It is rare, but it happens.
The other trigger has nothing to do with FDA. The day you stop exploring and start making design decisions somebody will later ask you to justify, your design history file needs to exist. That is usually earlier than either of the dates above.