For CXX flags specific for Release target, you should set
CMAKE_CXX_FLAGS_RELEASE
instead of
CMAKE_CXX_FLAGS
In your case you can use:
set(CMAKE_CXX_FLAGS "-Wall -Wextra")
set(CMAKE_CXX_FLAGS_DEBUG "-g")
set(CMAKE_CXX_FLAGS_RELEASE "-O3")
A more modern CMake approach (which I suggest, if you are using CMake version 2.8.12 or newer), is well described in this StackOverflow answerthis StackOverflow answer and involves the use of target_compile_options.