Electronic speed controllers designed specifically for brushed motors will only work with brushed motors. Likewise, electronic speed controllers designed specifically for brushless motors will only work for brushless motors.
So what's the difference between brushed and brushless ESC's?
If you're not sure how a motor works, go a head take a look at this short article that explains how standard brushed motors work.
In summary, as the armature rotates little tabs or brushes in the stationary portion of the motor make contact and provide voltage to the commutator which energizes the terminal for each pole or energized electromagnet of the motor. As the motor turns the voltage supplied to each pole changes.
As you can see, a brushed motor more or less runs itself mechanically as long as voltage is applied. To control how fast the motor spins a brushed ESC simply turns the voltage on and off very rapidly several times a second, this is called "chopping" the voltage...
To increase the speed, the ESC simply increases the amount of time that the voltage is on while decreasing the amount of time that the voltage is off. To slow the motor down, this process is reversed.
The key point in understanding brushed vs brushless ESC's is the fact that an ESC for a brushed motor doesn't care about the position of the motor. All it does is chops the voltage. Pretty straight forward.
Because brushless motors work completely differently,controlling a brushless motor is an entirely different process. Without brushes, the ESC has no idea what position the motor is in. It needs some sort of feed back from the motor to determine exactly when to energize each pole.
All brushless motors and brushless ESC's have three wires. Only two of the three wires are energized by the ESC at any given time. The pole that is not energized (coasting) at any specific instant will actually generate a small amount of voltage that is proportional to how fast the motor is turning.
This small voltage is used by the ESC to determine how fast and in what direction the motor is turning at any given time.
With this information, the ESC knows how to charge the electromagnets to keep the motor turning.