Daniel Gross  wrote an excellent article on potential job loss caused by the cooling of the real estate market called As the McMansions Go, So Goes Job Growth [NYT].  The decline in housing starts and building permits are perhaps an indication that the housing market is cooling and with that, the jobs they have provided may be affected.
Since the last recession in 2001, the real estate industry has provided 36% of all private-sector payroll job growth or 836,000 jobs. Real estate provides 9.7% of total domestic employment. There has been an unprecedented surge in membership to the National Association of Realtors over the past year as people try to make their fortunes as brokers in the housing boom [Matrix]. 
If we actually lose a few hundred thousand real estate related jobs and the Fed drops its concern about inflation with a weakening economy, it may be plausible that mortgage rates could actually trend downward as a result.