I agree with Massmomm - go see the new line and try stuff on but if you don’t find anything you like, be honest and explain that you are very picky and have a specific style that doesn’t work with her stuff. And alh’s ocmment of perhaps some designers you really like. It may be that your friend needs that kind of feedback to expand her offerings and sales.
Tough situation but I’m with those that say honesty is the best policy along the lines of, “I wish you all the best and I know you’re going to be successful but, unfortunately, my style and the things I like are different from what you carry”…OK, I read that and it’s not perfect but I’d start there and work on it. I believe others have had better wording. And I agree that I’d stay away from the price point issue. In any case, assuming she’s able to stay in business, the issue won’t go away so you might as well bite the bullet and address it.
There are other ways you can support her than with your wallet.
If you have friends or coworkers that you think might like her shop (don’t put your own bias into the conversation) tell them about her place - and tell them “tell her that Maya told you to stop by!” - they may end up being better customers than you would be anyway! 
If her business has a FB page share it on your FB and just say “a friend opened a new boutique in town - check it out” - will help create awareness, show support and again, maybe land her a new customer that might not have known about her place.
No harm in looking but I would tell her you don’t want to waste her time. And if it’s true, you could tell her you are rationalizing your wardrobe and trying to simplify it to a smaller number of items you really love. In that vein, you aren’t adding to your wardrobe and/or are doing it very selectively.
If her things aren’t your style, it’s fine to remark on the beauty, quality, etc., noting that it isn’t something you’d get enough use from to justify the purchase.
I can think of dozens of boutiques I love to visit for their merchandise and the way it’s displayed – both reflecting talented owners – where I would almost certainly walk out empty-handed. Too dressy, too business-y, too trendy for my lifestyle. That’s what happens when a collection is highly curated.
You could just not go back to the store to see the new items. She will probably get the message.
If she pursues the matter further, then you could tell her that her clothes don’t suit your style, but that you wish her well in her business, etc. I like the wording suggested by many of the previous posters. It need not be hurtful; she has to realize that people have different styles. I would avoid commenting on the cost of her items, though.
I would be furious if a “friend” tried to presume on our friendship to give me the hard-sell for things I didn’t want to buy. I also question this person’s judgment, if she continues to press her case even though you haven’t bought anything from her and don’t seem to want to buy anything she sells. “You buy expensive things and I sell expensive things so you should shop in my store,” is offensive, really. I’d respond to all her solicitations with a simple, polite, “No thank you” until she got the message, and I’d refuse to listen to a sales pitch.
We had good friends who tried to pressure into start selling for Amway. “You’ll make so much money you can retire early!” “Uh, we don’t want to retire early…” Ugh, it was so awkward.
I think you’ve gotten great advice already on how to handle things related to your own purchases, but one question came to mind. How is the store doing? Is her taste just not yours or is she really bad at her selection? If nobody likes her expensive items, then maybe some friend needs to suggest that she perhaps bring in a focus group of her targeted clientele to help her hone her selections.
Whatever you say, please don’t say “I’m just very picky.” That is the equivalent of saying “your selections aren’t good enough for me,” and quite insulting, IMHO. Even though there is no intention to insult.
Saying “it’s lovely but it’s not my style, alas” is fine, I think.
I also like the ideas suggested by @abasket regarding helping her by liking her FB page and so forth. If she has something like a little evening preview event with wine and simple refreshments, you can invite other FB friends to the event, or even invite along some friends. A lot of women love events like that.
All of these comments are super helpful and made me think about why I don’t “ shop local” even if I do like the clothes or other items. It’s the return policy. I shop mainly at Nordstrom and Bloomingdales and both stores give me at least a year to return. So if something sits in my closet unworn for a long time after I buy it I often return it. And they stand behind their stuff. I had a beauriful expensive sweater with lace at the sleeves that tore after I’d worn it 3 times. I brought it back and told them how often I’d worn it and they gave me my money back.
i understand why a boutique owner cannot afford to have such a generous return policy and that they don’t like standing behind merchandise that was worn a few times, even though they maybe should. They probably just can’t afford it.
Maybe I’ll tell her some version of my reasons , that I understand that she can’t honor the kind of return policy I need
As someone who owns their own business (not a boutique and not an MLM), I will say that I value feedback from my friends. I am always looking to improve what I offer. If your friend is like this, I do think you should give her some honest feedback. Tell her that you think it’s great that she is running this boutique and that you are willing to listen to her vent and want to support her. Then gently tell her that at this time that just doesn’t include shopping at the boutique because the clothes there just aren’t a match for your style. She may be surprised to hear that. She may have questions. Or she may get mad. (you never know) Hopefully it works out for you!
I think it’s fine also to be more specific… “I tend to like understated (versus tight/blingy) or traditional (not trendy) or colorful . . . .”
I agree with the above poster that her business might not be doing well and she is trying to drum up sales. Double agree it is very uncomfortable!
All these comments about @maya54 's style got me to thinking about my “style” which I might best describe as “clearance rack chic”. 
@maya54, I thought I was the only one who likes a more liberal return policy.
And yea, I don’t buy as much at these boutiques because of the return policy.
At the one, it has to be returned within 14 days and only for credit. I bought an item and the very next day, she had a store wide sale 25% off. I went back for an adjustment and she didn’t want to give it to me. I was annoyed and let her know that I wasn’t inclined to shop there because of that. Her feelings were that she could have sold it to other (i.e. better) customers earlier as it was an in demand item. She finally relented and I did buy another item to cover the cost of the return. The owner also likes to push my fashion choices as I like to say. I get home after buying something and I look ridiculous. But I feel stupid returning so I keep it and wear once or twice. And it’s not comfortable. I hardly ever buy anything from there now.
I want to buy stuff, come home try it on and see what I have in my closet. If it isn’t right, I’ll return. Much easier to do when it’s in a store where it’s not personal.
But I thought I was the only person who thought this way and I’m glad you said it.
Just don’t go to the store and don’t say anything, someone you talk to maybe once a month is not worth all this handwringing. IMO every time you go in to the store you are making this worse. Why DO you ever go in?
Why do you need to explain anything? Just stop going to her shop. She will get the message.
I’d buy something - a scarf, a hat, a belt - who cares if it isn’t your style? Obviously. the price tag is not an issue for you, and there is always somebody, somewhere who would appreciate a nice gift.
My point is, for a few hundred bucks, you can save a friendship, or at the very least, hurt feelings, and bring joy to somebody else, somewhere down the road.
And please tell us the $$$$$$$ of your bag, LOL!
Lol. Not a Burkin. “Just” this https://shop.nordstrom.com/s/valentino-garavani-vitello-rockstud-lambskin-leather-shoulder-bag/4910691?origin=coordinating-4910691-0-3-PDP_1-recbot-also_viewed&recs_placement=PDP_1&recs_strategy=also_viewed&recs_source=recbot&recs_page_type=product&recs_seed=4570086. Which I do think qualifies for all those $. My prior most expensive purse was $79.
Gorgeous purse @maya54! IMO much better than the Chanel purses you see
@maya54 I love that purse, it’s gorgeous! Lambskin is my favorite leather. FYI, I was in Marshall’s this week - if you can believe it, they had Rockstud shoes for $450 - not my size, unfortunately, but you could get a pair to match your bag if you were fortunate to find your size.