Not everyone has an email.
A businesswoman once proudly shown me a dumbphone and said that she does not have a personal email, only a corporate one. That persuaded me for a while, until I learned that her husband was prosecuted for money laundering around the same time. So yes, not everyone has email, but those who don't are few and they have very good reasons.
Email is unsecure
Aside from spam and automated emails, pretty much all email typed interactively in an email client is encrypted between the sender and the receiver.
We could have a long technical discussion here about the opportunistic encryption of STARTTLS or about the market share of Google, Microsoft and Apple, but the reality is that is is protected for all practical purposes that matter to ordinary people. That is, it is impossible to view and modify in transit emails that are written by individuals.
You can impersonate anyone in an email
Long gone are the days when you could send a mail from gates@microsoft.com from your personal computer. To start with, port 25 is probably blocked for sending at your ISP. Then, even if you managed to send an email, it will be probably rejected as coming from a residential range of IP addresses. But even if you send a mail from Amazon SES, then the receiving SMTP server will use SPF and DMARC to check if Amazon SES can send emails on behalf of @microsoft.com and it won't.
An email can not be used in a legal dispute
IANAL and there are many technical solutions that work towards legally binding emails e.g. eIDAS. However one little known accidental feature of DKIM email signatures allows to check if the email was indeed sent by the sender and was not tampered with. If the sender used a well known entity to send email, e. Microsoft or Google or even GMX, then this proof is irrefutable.
Magic links in email should not be used for authentication
Many people, me included, do not bother saving the passwords for the websites they use only occasionally. Instead, they just put a random string into the password field, forget it and use the recover by mail link to access the same website later, if needed. This is a rational behavior for many reasons, but the main one is that websites tend to reset passwords once in a while anyway. Either because of a breach or platform change or policy, so the chances of being able to reuse the same password years after are slim anyway.
There are services operating using magic links only, although this is a tricky system to maintain due to adversarial behavior of industry. This NH thread gives a good overview of challenges and none is really about security.