I have this one iMessage conversation that (always) takes about 1.5 seconds to load when switching to it. Check the delay between the contact list marking it as the active conversation and the content actually changing.
There is nothing special about this conversation. No large media in it, or anything else that could explain it. This has been going on for almost a year now, and it keeps happening after restarting (both the app and the whole computer). In fact, this bug even survived me switching to a completely new computer, which I did not restore from backup. I just logged into iCloud and it downloaded my iMessages from the cloud – and as soon as the messages were back, the behavior was back as well.
Of course, the conversation opens as instantly as any other on my iPhone.