9
$\begingroup$

The June, 2015 self-evaluation didn't seem to go over well and we are rethinking site checkups. So I thought this would be a good opportunity to try an experimental self-evaluation.

Below, I've listed 5 questions chosen at random from the last quarter that have received at least some anonymous feedback. I also included two scores*:

  1. How helpful the page seemed to regular users.
  2. How helpful the page seemed to anonymous visitors.

The final number is the number of total views the question had received at the time of sampling.

If you'd like to participate, please copy the question list into an answer for evaluation. Exactly how you evaluate the questions and answers is up to you, but it would be useful to:

  • Find better answers on the internet (if they exist).

    Exactly what "better" means is up to you, but the criteria might include:

    • correctness,
    • understandability,
    • authoritative (e.g. citing official sources),
    • clean formatting, or
    • easier to find with a search.
  • Figure out why regular users and anonymous visitors have different opinions of the usefulness of the page.

    The most common reason for low visitor scores is low views, which limits the number of anonymous users who provide feedback. The score accounts for that situation by assuming visitors don't value the question highly. It's not a problem if some questions are so niche that nobody outside of the community reads them. But chance visitors who find answers via search are the best source of new contributors.

    More interesting: sometimes visitors disagree with the site's community about the usefulness of a question and its answers. Obviously we can't know why people who can't leave comments might differ from current users, but we can make an educated guess.

  • Fix any problems you notice and describe your changes.

    After doing the above analysis, you will be in good shape to edit titles and tags to help Google direct searchers to the question, clean up formatting and grammar problems, and link to authoritative sources. There's no need to wait; go edit.

The goal is to use the sample to reflect on how the site is progressing in terms of "making the internet a better place for people to get answers to their aviation questions". If this site already does a good job, that's wonderful. If there are some things that need fixing, please talk about that. In either case, this evaluation mostly stands to improve this community's understanding of itself.


  1. What is the maximum age limit for getting a pilot licence? (51:15 234)
  2. Where does ice collect first on a wing in flight and why? (85:34 453)
  3. Why is the landing area of an aircraft carrier angled to port side, not starboard? (94:67 4,439)
  4. Why do fighter jets land faster than Jumbo commercial aircraft? (91:44 3,693)
  5. Why are contrails not visible from the cabin? (93:21 1,168)

Footnote:

* The scores are actually the lower bound of Wilson score confidence interval for a Bernoulli parameter, which is a measure of how likely a given set of upvotes and downvotes is overall positive. For the first score, I used the sum of all votes on the question and all answers. For the second score, I used anonymous feedback. Please see the query I used for implementation details.

$\endgroup$
3
  • $\begingroup$ Experimental? I'm not sure how I feel about that! $\endgroup$
    – voretaq7
    Commented Jul 9, 2015 at 16:12
  • $\begingroup$ Seriously though, I think this is a cleaner way to do evaluations. I do have to question the validity of the anonymous feedback data because of the limited sample set though (only one question had enough feedback to have even a chance of statistical relevance). It might be difficult to draw any meaningful "Community-vs-Anonymous" conclusions based on this data set. $\endgroup$
    – voretaq7
    Commented Jul 9, 2015 at 16:17
  • $\begingroup$ Slightly tweaked query - I removed the "top 5" so I could get a sense of the results because the random ordering from newid() is sketchy: It picked what I'd consider a good representative sample in your original query, but when I cranked the anonymous feedback parameter the sample set wasn't anywhere near as diverse. $\endgroup$
    – voretaq7
    Commented Jul 9, 2015 at 16:33

2 Answers 2

8
$\begingroup$

Short Version: My impression of the selected questions & their associated answers is generally positive. I'm not thrilled with all of them, but I don't think any are terribad.


OH GOD SO MUCH TEXT MAKE IT STOP version:

1. What is the maximum age limit for getting a pilot licence? (51:15 234)

Subjective Overview

The only thing "wrong" with this question is that it doesn't specify what country's regulations they're asking about or what their flying goals are (Private, Commercial, or Military), which makes it difficult to answer definitively. (Unfortunately these are things the asker needs to fix - there's no obvious way to fill in the blanks from their question & profile.)
If the asker returns to fix the deficiencies noted this is a "good question" - it's one a pilot or someone directly involved with aviation probably wouldn't ask, but someone interested in starting a career as a pilot would certainly want to know if they're age-limited (particularly if they're considering a career change later in life).

"Helpfulness"

There's not enough data for a statistically meaningful assessment of how "helpful" this question is, and the question itself isn't detailed enough to evaluate the answers for "correctness".
The answers that were posted before the question was closed do attempt to give some insight that might help point the asker or subsequent visitors in the right direction for their own research (this one in particular stands out).

Overall I would say the answers here "make the internet a better place", though they're not as solid as I think most of us would like to see.

Versus other sites

This question is asked constantly on forum sites. The answer almost universally points to the FAA regulations (because those sites tend to be USA-focused).
We do not link directly to the FAA regs, but that's because the community made a conscious decision NOT to limit the site scope to the USA and instead to try to determine what regulations apply in the poster's specific situation.


2. Where does ice collect first on a wing in flight and why? (85:34 453)

Subjective Overview

This is the sort of question we'd expect from an advanced private pilot or an instrument student - about the only thing missing from the question is a link to the "report" which would give people looking to answer it a starting point for their research and explanations.

"Helpfulness"

There are enough views and votes here that I think the feedback on the question and accepted answer can be deemed "overwhelmingly positive". There aren't enough anonymous votes to definitively say whether the internet at large agrees with the community, but as there are no downvotes from either group there seems to be a consensus of opinions.

The answer itself uses the broad wording of the question to lead into a good discussion of the subject at hand (the various types of ice, when they form, and where they tend to appear). It also links to a number of respected and credible sources for additional reference and research, so folks wanting a more in-depth explanation have somewhere to continue looking. About the only thing it doesn't cover is "Does ice accumulate on the inboard part of the wing first, or the outboard part?" (which is something the sources also don't address - I imagine there are a number of factors that would contribute to that answer on any particular airframe, so they concentrate more on front/back position of the ice accumulation & its affects on de-icing systems).

Overall I would say this question & answer makes the internet a better place.

Versus other sites

The answer here is a consolidation of information available from other resources (which are linked to). While we're not creating "new knowledge" with this answer we are presenting it in a quick, condensed form & providing links to source sites.
There's value in the condensed form of the answer that we're presenting, and providing the "expanded" answer via external resources seems to be the best way to handle discussions about icing: A full discussion would quickly become too dense for many readers.


3. Why is the landing area of an aircraft carrier angled to port side, not starboard? (94:67 4,439)

Subjective Overview

It is not surprising that this is the most-viewed question in the sample set: Anything about military aviation, commercial aviation, or timely questions about aviation accidents tend to accumulate views quickly, and maintain a high level of interest. They're also Google Candy: This question is the second result when I ask Google Why are aircraft carrier decks angled to the left?

Because of this popularity this question has wound up "Protected".
This is rapidly becoming our standard practice for such questions.

The question itself is clear and well-scoped.

"Helpfulness"

The overall impression from both the community and the anonymous users who have voted on this question is positive, and there's a good correlation between the two.

Sadly we don't have many (any?) naval aviators or aircraft carrier designers around to give definitive answers on this subject. The top 3 answers give sound, plausible reasons though, and they cite at least one compelling source. If I were to find fault with this question & the answers it would be that "the answer" requires reading at least 2 posted answers to get all the details. This is a common problem with questions where we don't have a de facto subject matter expert on the site to give a detailed write-up.

Overall I would say the answers here "make the internet a better place", but I wouldn't hold them up as an example of our best work.

Versus other sites

The other site I would compare us to here is Quora - the #1 response in my Google search.

  • The Quora answer is better organized than ours, but also only concentrates on one aspect (flight safety & aborted landings).
  • Our answers include references which provide the history of the Royal Navy's rationale behind angling the deck to the left as opposed to the right, which isn't covered in the Quora answer or comments.

I would say it's pretty much a wash.


4. Why do fighter jets land faster than Jumbo commercial aircraft? (91:44 3,693)

Subjective Overview

Another military question, and another highly-viewed one.
This is one of those questions where the answer is sort-of self evident if you're a pilot or have an understanding of how lift is produced by a wing, but it's a good general-audiences question.

"Helpfulness"

The accepted answer gives the "Why", but is perhaps a bit too technically dense for a novice (how many people off the street know what "wing loading" is?) - it at least puts the answer within easy reach though.

The other answers and comments…sort of muddy the waters more than they help. At least one seems to confuse Effect and Cause to some extent.

Overall I would say the existence of this post makes the internet a better place, but again I wouldn't say it's the site's best work.

Versus other sites

There don't seem to be any directly comparable questions elsewhere on the internet (at least from cursory Google searching: Our Q&A is the top result, followed by several irrelevant results).


5. Why are contrails not visible from the cabin? (93:21 1,168)

Subjective Overview

This is another good "general audience" type of question - something many people have probably wondered sitting in the back of an airliner.
It could be argued that the question is more a physics one than an aviation one (really the only thing contrails have to do with aviation is "you see them behind jets"), but I would consider it appropriate for the site.

"Helpfulness"

The top two answers provide the "How" and the "Why" behind the formation of contrails, and explain why they are (usually) not visible from the cabin. they also provide an example of at least one case where the contrails were probably visible from the cabin (exterior photo, so we can't be certain).
The third answers is "more a comment than an answer" and is a good example of the most fundamental misunderstanding of "This is a Q&A site" that we see, but it's not bad enough where I would delete the answer: There is some value in the examples it provides.

**Overall this question and its answers "make the internet a better place", while there are lots of web pages about contrails the explanation here is distinct from what I've seen on other sites and covers a slightly different aspect of the subject matter.

Versus other sites.

I refuse to search for contrail-related things on the internet on the grounds that I will wind up reading some chemtrail conspiracy theory and throwing my coffee mug through my office window. - Sorry, you're on your own with this one :)

$\endgroup$
4
  • $\begingroup$ Thank you for responding so thoroughly. This is exactly the sort of evaluation I was hoping for. I think I will try this experiment on another site, but rephrase the prompt to go for answers like this one. Did you find the scores at all useful? Maybe it would be better just to include views at the time of sampling. $\endgroup$ Commented Jul 9, 2015 at 18:42
  • $\begingroup$ I think we have at least one Naval aviator, they just haven't posted on the carrier flight deck question. $\endgroup$
    – fooot
    Commented Jul 9, 2015 at 19:48
  • $\begingroup$ See, all I did was read your review of the question I wrote. I'm feeling pretty proud for giving the impression I must be an IFR candidate since, you know, I have all of 3 hours of seat time to my name... :D. All the same, I'm assuming that's the audience this would generally apply to, right? $\endgroup$
    – Jae Carr
    Commented Jul 9, 2015 at 21:01
  • $\begingroup$ @JonEricson The scores are kind of misleading because of the low volume of anonymous feedback. When I played with the threshold of anonymous feedback in the query they became a lot more useful as a metric. For low-volume questions (upvotes - downvotes)/total votes might be a better metric, but that's a pretty blunt instrument. $\endgroup$
    – voretaq7
    Commented Jul 10, 2015 at 2:58
5
$\begingroup$

I'd really like to help, but I'm not quite following this... Any chance you can put the experimental self-evaluation in idiot proof terms so I can understand it? I'm really not sure what we're being asked to do.... I'm making a proper effort at understanding the query and the data it's spitting out but... Well, even when I do figure that out I'm not sure what ought to be done with it.

My biggest confusion, I think, is the data set itself. It seems like 4 of the 5 questions don't have enough data to infer really anything about how anonymous users felt about the questions, as none of those 4 have more than 3 votes in either direction. Doesn't it stand to reason that the questions selected should all have votes like unto "Aircraft Carrier" question which had 33 votes in either direction from anonymous users?

Another point of confusion is the first criteria for evaluating the answers: Better in what sense? As in more accessible? Or as in more technically accurate? (which may have the exact opposite effect of the intended effect...)

Though point 3 seems like a pretty useful start of a discussion... Though, as I said, I'm not sure the best questions were pulled for having that conversation.

$\endgroup$
4
  • 1
    $\begingroup$ Probably the missing piece of information is that I picked the sample randomly. As long as the page had at least one anonymous feedback vote and was asked in the last 90 days, it was a candidate. I didn't work very hard to generate an interesting sample (though that might be interesting). I've tried to address your other points with an edit to the question. It seems to me that I probably tried to bite off too much for this particular experiment. $\endgroup$ Commented Jul 9, 2015 at 18:32
  • $\begingroup$ @JonEricson lol, if it's any sort of encouragement, I think this is really on the right track. I much prefer it to the old system anyway... $\endgroup$
    – Jae Carr
    Commented Jul 9, 2015 at 20:58
  • $\begingroup$ @JonEricson "random" is kind of important - I think there's a lot of promise in this experiment. The big variable that I think needs tweaking is "how much anonymous feedback before we include the question in the list of candidates?" - something that's probably going to be site-specific (depending largely on traffic volume) $\endgroup$
    – voretaq7
    Commented Jul 10, 2015 at 2:51
  • $\begingroup$ @voretaq7 what if you just said it had to be in between the 30th and 70th percentile of vote getters? Mr maybe sticking around 50th percentile doesn't matter and just say it needs to be 50th percentile and above? $\endgroup$
    – Jae Carr
    Commented Jul 10, 2015 at 3:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .