I've been in one house party, but I must admit I hadn't heard the term until I was asked, at the tender age of 19.
My understanding is it's your close friends who you want to have a role but don't/can't, for whatever reason, want to make bridesmaids. Often it's reflective of the size of the wedding. The wedding for which I was in the house party was small with just the bride's sister as maid of honor. Those of us in the house party did things like serve the punch (my role), cut the cake, etc. I got the impression it was a Presbyterian thing.
But I found a link to a site that describes it in more detail:
The House Party: Roles and Responsibilities