The primary control for determining how the sea on your world appears is ..
*settings → Lanscape → Sea Rendering Mode
This setting controls the general appearance of your sea (whether it is a plain colour or textured, whether it is semi-transparent etc..)
First of all you need to set *settings → Landscape → Sea Rendering mode
to one of the modes that indicates wave effects are applied. (5 or 6)
Once thats on, the amplitude, wavelength and general behaviour of the sea is controlled using..
*settings → Landscape →
These values are combined in magically random mathematical ways to determine where each point in the sea is at any particular time. (For 'magically random mathematical ways' read 'Mit has no idea how his code works' ;] ).
Wave Range Mod
Wave Height Mod 1
Wave Height Mod 2
In theory, 'Range mod' controls the wavelength (i.e. a high value creates waves with peaks close together) and the other two are combined to create the height of the waves.
Couple of final 'advanced user' notes..