That was my exact experience with the flu when I had it many years ago. Not sure if our N = 2 is truly representative.
I second keeping a couple of those lateral flow tests on hand and calling your doctor or going to an urgent care if a test is positive.