Did you pay too much for your mortgage? If you’re like millions of Americans, the answer is probably yes — and that means you may be throwing tens of thousands of dollars of your hard-earned money at the bank, when you might not need to.

A report released Tuesday by the Consumer Financial Protection Bureau finds that almost half (47%) of Americans don’t shop around for a mortgage when they purchase a home. “Consumers put great thought into the choice of a home, but the mortgage process continues to be intimidating,” CFPB Director Richard Cordray said in a statement.

Number of lenders Americans seriously consider before applying for a mortgage

Source: Consumer Financial Protection Bureau

If you don’t shop around for a mortgage, you’re probably leaving free money (and a lot of it) on the table. “Interest rates can span more than half a percent for a conventional mortgage for borrowers with a good credit rating and a 20% down payment,” says Sam Gilford, a spokesperson for the CFPB.

While half a percent may not sound like a lot, it can be a more than $25,000 mistake for the average borrower (as of November 2014, the average price of a home sold in the U.S. was about $321,800, according to data from The Census Bureau), who takes a mortgage that’s half a percent higher than one he could have gotten by shopping around.

Say a borrower accepts a 4.5% interest rate instead of a 4% interest rate on the average home (a sale price of $321,800 and a down payment of 20% means he borrows a total of $257,440). If he gets a 30-year fixed rate loan at 4.5%, he’ll pay a total of $212,148 in interest; for a 4% interest rate, he will pay just $185,021 — a difference of more than $27,000.

For those who buy a home that costs more than average — or who put down a smaller down payment than 20% even on an average home — the results may be even more grim. For example, a person who gets a $500,000 mortgage would pay more than $412,000 in interest over the life of his 4.5% 30-year fixed rate loan, which is roughly $53,000 more than with a 4% rate.

To be sure, many people who don’t shop around may get the best rate anyway — or at least close to it. Others get a mortgage that may be a little too costly, but will later refinance and save themselves money. And still others will sell their home well before the 30-year loan period is up, so they end up paying less in interest.

Still, experts say it’s worth shopping around, as even 1/10th of a percentage point can mean thousands of dollars in extra payments to the mortgage company over the life of a loan. Luckily, doing so is relatively easy. Before shopping around, Greg McBride, the chief financial analyst for Bankrate.com says that you should pull copies of your credit reports from each of the three major credit bureaus (you can get these for free atannualcreditreport.com), consider what type of loan makes the most sense for you (see MarketWatch’s “How to Get a Mortgage” guide) and figure out how large of a loan you can afford (there are dozens of online calculators that can calculate your monthly payments and more).

Once you’ve done that, McBride says that you should get quotes from your local bank and credit union as well as online and apply with up to three lenders on the same day. (Kathleen Campbell, the founder of Campbell Financial Partners in Fort Myers, recommends using Bankrate to check mortgage rates, and considering online lenders like Quicken Loans, CapitalOne 360 and Pentagon Federal Credit Union.) Finally, “compare all lender fees and rates, negotiate to get the best deal, and select the best offer,” McBride says.

This story was originally published Jan. 13 on MarketWatch.com.