The alert state indicates that guards are now on the lookout for the player. They are aware of his presence, and are suspicious.
How to Enter Alert State
They also might suspect him if he fires a dart at them and misses, or attempts some other sort of attack and whiffs.
How to Exit Alert State
To exit an alert state, simply hide for a long enough time or bribe a town crier.