By early Friday, the fire had spread to the fringes of neighbouring Santa Barbara city, where at least 20 homes were damaged, officials said.
Local authorities say more people could be forced to flee if the fire spreads, although winds were reported to have died down by mid-morning.
Those injured were treated for burns and smoke inhalation.
Southern California is on high wildfire alert this weekend due to hot weather, drought conditions and forecasts of strong winds blowing in from the desert.
A year ago, 30 wildfires burned during one week in the southern part of California, destroying about 2,000 homes and forcing a record 500,000 residents to evacuate.