Professional Look
For the layout, learn and use a page layout program. Word is NOT a page layout program; it's a word processor with delusions of page layout capability. Word is excellent for doing the original files, tho'; generate the RTF files using the styles system in word or another styles-using wordprocessor (I use Pages; I used to use Appleworks and Word Perfect)
True page layout programs are more powerful at graphics handling and text manipulation than Word... but generally have limited text editing capabilities.
The precise control and handling features of true page layout programs allow far better control of look and feel, underlayed and overlayed graphics, more powerful style options.
There are several pay-ware options; Indesign is pretty standard. Scribus is the free opensource answer; it's very similar to Pagemaker in scope and features.... since I used to use Pagemaker (at work, back in 1996-98), it's very familiar.
Art
If using electronic art, try to get it in vector versions (like SVG), rather than raster images (like jpeg, tiff, gif, or bmp). The vector images scale nicely, and you can use them at any size; rasters need conversions and should be either 300dpi or 150dpi when used, and should be scaled in integer divisions downward only.
Pick One Style
Don't mix-n-match art styles. If using paintings, use only paintings. If using line art, use nothing but line art. If using Poser models, make certain that the renders are consistent and very high quality.
If using multiple styles, at least be consistent. Digital Paintings for title and chapter title pages, with line art for tech drawings, or Comic Book art for characters and digital art for tech. Plan your art, don't just find your art.
Using Raster
300 DPI means large files. It's also moderately good for printing professionally; most people won't be able to see the differences between 300 and 600 DPI print jobs, so unless going for high end work with lots of details, it's not worth the extra for a print based product.
150 DPI is standard "Web Resolution" for PDF. It's visibly grainy on screen when viewed at more than 200% (most screens run about 75-100 DPI, with some rare monitors as much as 200dpi). It prints fairly well, tho', and except for maps, is usually sufficient.
I any case, you want your art oversized. if asking for a scanned page piece, you want it scanned at 300 DPI or more; if you can handle the drive space, go for 1200dpi scans. Then, having locked the source file, you make reduced DPI versions from the original scan (never from each other; successive conversion errors creep in and can make it anti-alias wrong). If at all possible, make your size choices an integer division. And never scale up a raster image. Always resample from the original and only to make it smaller.
Professional Quality Content
For the game design elements, playtest. Then playtest some more.
Revise and rework, then playtest again.
Realize that, using FATE, you have more to do than just a few changes to the SRD; make certain that examples and commentary are kept to current rules. Not much more annoying than examples that are in violation of the published rules.
TOC and Index
Your Table of Contents (TOC) is vital as it is the first place people will look for where to find things. But even a comprehensive TOC is no substitute for an index. Any game past about 15 pages should be provided a TOC.
The TOC should have every chapter, and each major division of a chapter. Each specially named rule should also be listed.
The index is far harder to get right, but more important. Every game term should be in the index. So should any named rules, common situations, and generic gaming terms which may be alternate names for chapter titles (Like Combat leading to the chapter Violence, Character Generation leading to Creating Characters).
PDF Specific niceness
Link up your TOC and Index. It's a pain, but it makes it SO much more useful.
As SevenSidedDie notes, Make certain your metadata is correct. If need be, manually edit the metadata. There are free and paid options in all price ranges for editing metadata; the better ones also can do other editing on the PDFs. The metadata is what various ebook readers use to categorize the PDFs.
Other Electronic Formats
Generally, 3 other electronic formats are popular - Kindle, ePub, and mobi. These do not preserve layout, and have limited formatting. If you decide to do a version for these, you need to test it at multiple sizes, and check the reflow to not do violence to the contents.
Microsoft RTF and Word Doc are poor choices for electronic distribution. While there are multiple readers for almost every currently sold computing platform (including Windows, MacOS, Linux, Unix, iOS, Android, ChromiumOS, and Blackberry; only online choices for ChromeOS), rendering on any given one can be flaky. If you do distribute a Doc or RTF, use no hard page breaks except at end of chapters. Don't bother numbering. Use only whole number point sizes. Use only the PDF-standard's default fonts - they're pretty much supported on everything.
Don't use non-breaking spaces nor tabs - while they're standards, they often do unkind things with ereaders, and exactly what is unpredictable.
Metadata is even more important with all of these types. Word Doc and RTF both include provisions for metadata, but many people are unaware of this.
