Windows can not reliably remember your default audio output device. Whenever a “new” audio device is connected, Windows automatically switches to that device instead. Not “new” as in first time the audio device is seen on your system, but “new” as in every damn time it is “plugged in” – for example when you power off a HDMI-connected display and then turn it back on again.
There is no way to disable this “feature”, the oldest occurences of people desperately trying to find a solution are about eight years old. There is tons of threads all over the internet about this insane behavior. It even happens when updating your GPU drivers, as that makes those HDMI ports “new” to Windows as well for some fucking reason.
But what can you expect from an operating system that ships the above-pictured system preferences UI mashup and thinks anything about that is even remotely acceptable or normal. (In case you are blissfully unaware: The window on the right is what you get when you click “Advanced” somewhere in the window on the left. The window on the left is the new UI, but half the system still uses the old Windows XP era style UI, because… well, no idea why. Some settings can be changed in both parts of the preferences, some only in one of them. Absolutely bonkers.)
The “solution”, in classic Windows style, is to hack something together yourself with AHK and a 3rd party tool that feels like it time travelled to my aid from the 90s. I’m glad I’ll be able to automate it somewhat, but just look at that website. Look at the first two options in the usage examples. OPEN CD-ROM DRIVE! That kind of tool is what you need for setting your default audio device reliably on Windows 10.