Idea and Ontotree Pages:
Frequently Asked Questions

1. What does it mean to say that two ideas are highly related?
2. What does it mean to say that one idea is more specific than (or more general than) another?
3. If I judge that two ideas are totally unrelated, do I need to assess their relative generality?
4. How should I decide the relatedness or relative generality of two ideas?
5. How do you generate the lists of candidate pairs I am seeing?
6. How many evaluations do I have to do?
7. Is it ok if I leave some pairs blank?
8. What if I’m not sure, and need more information to decide about a possible relationship between the two ideas?
9. What if I feel sure that two ideas are semantically related, but I can see an argument for either direction of their relative generality?
10. What if one of the terms is ambiguous?
11. What if I know how something is commonly classified, but I personally disagree with that classification?
12. What happens to my evaluations once I submit them?
13. What if someone else’s feedback disagrees with mine?
14. What does the “SEP” link do?
15. How should I interpret “and” and “or” categories?
16. What does “add your own” do?
17. What do the statistics on the myspace page mean?
18. Why do some categories not have any instances on the ontotree browser?
19. Why do some categories not produce any pairs for evaluation on the idea evaluation pages?
20. Can I review/change my previous evaluations?
21. What if I disagree with the categorical structure of the ontotree, or feel that there are important areas being neglected?
22. Why hasn't my feedback added any new instances to the ontotree?
23. How hard should I think about idea pairs which seem odd?
    
    Questions still unanswered? Send them to inpho <at> indiana <dot> edu.

1. What does it mean to say that two ideas are highly related?
   
   Two ideas are highly related if consideration of the first idea “naturally” or “necessarily” raises consideration of the second (and vice versa) in the context of that specialization of philosophy.  For example, suppose you are writing a somewhat comprehensive book on the more general of the two ideas; how strongly do you feel that you should mention the more specific idea in that book?  Or, suppose you are writing a paper on the more specific idea (or the two terms are the same level of generality); how strongly do you feel you should locate your project with respect to work on the other idea?  If two terms are highly related, it is strongly recommended to consider issues concerning idea B when dealing with issues concerning idea A (and vice versa).  If two terms are related, then A and B are often, but perhaps not necessarily, mutually relevant.  A “neutral” vote might indicate that you can see how the two ideas are commonly or distantly related, but explicit mutual treatment would not be expected.  A vote of “unrelated” or “highly unrelated” should indicate that you would be surprised to see one deal seriously with idea B in the context of idea A (and vice versa).
   
   
2. What does it mean to say that one idea is more specific than (or more general than) another?
   
   One idea is considered more general than another if it is considered in a wider range of contexts, or it plays a more important role in that specialization of philosophy.  For example, the most general ideas are likely to be the topics of numerous survey books and courses, and are likely to constitute areas or problems with which a philosopher could be occupied with for much of her career.  The most specific ideas are likely to be mentioned only in a few very specialized papers, and are likely to be dealt with directly only by a small number of philosophers.  Most ideas will fall in the middle of this scale, and that is why we should consider the relative generality of the two ideas.  If they are highly related, how likely is idea B to be a sub-topic, or minor issue, reviewed in a book about idea A (and vice versa)?
   
   
3. If two ideas are totally unrelated, do I need to assess their relative generality?
   
   No; if you indicate that two ideas are highly unrelated, the generality dropdown box will be removed.
   
   
4. How should I decide the relatedness or relative generality of two ideas?
   
   We want to acquire information about how philosophers organize the semantic space of a discipline in their particular areas of expertise. Thus, you should evaluate the semantic relationship between the two ideas only in context of the broad sub-specialization of philosophy in which the category is classified (e.g. metaphysics, logic, philosophy of mind, ethics, etc.). 
   
   You should also not require or expect these semantic relationships to hold domain-generally. For example, while “connectionism” is an idea that might naturally be subordinated to the heading “philosophy of mental content” in the context of philosophy of mind, in the context of the discipline of computer science, the direction of relative generality would be reversed. 
   
   
5. How do you generate the lists of candidate pairs I am seeing?
   
   The lists are generated by combining a statistical measure of the semantic relatedness of the terms in the SEP with a measure of their relative generality.  The relatedness score is computed by looking at the number of times the terms co-occur throughout the encyclopedia, either in the same sentence (weighted more highly) or in the same article (weighted less).  The relative generality score is generated by taking a measure of the terms’ relative entropy—in short, do the terms occur just a few times in a small subset of documents, or are they very common words that occur frequently throughout the corpus?  The assumption here is that the higher a term’s entropy, the more general it will be in the encyclopedia; and the lower the term’s entropy, the more specific it will be.  These scores are then combined into a single measure, which allows us to generate a ranked list of candidate hyponyms (more specific and highly related) and hypernyms (more general and highly related) for the target term.  It is this ranked list that is displayed to you for evaluation.
   
   
6. How many evaluations do I have to do?
   
   You can do as many or few as you would like, and feel free to stop at any time. (But remember to click “submit” when you are finished in order to log the evaluations you have made!) 
   
   
7. Is it ok if I leave some pairs blank?
   
   Yes, you can leave some blank, although if you have time, you are encouraged to investigate difficult cases. 
   
   
8. What if I’m not sure, and need more information to decide about a possible relationship between the two ideas?
   
   We have included a link, appearing as “SEP”, near each evaluation.  This link will search the SEP with a search string that is a conjunction of the two keywords.  The articles found by this search are likely to be the articles that provided the evidence responsible for the pair’s appearance on the list.  Reading a few of these articles—and especially searching for passages in the articles where the terms co-occur—can help you determine whether the two ideas are genuinely related or if the co-occurrence is mere coincidence.  You are encouraged to confirm or disconfirm the hypotheses in this way especially for terms which initially seem like odd pairs to you—we have come upon several “discoveries” that we would not have otherwise been aware of in this manner.  Examples include anaphora -> propositional attitudes (linked via Frege’s theory of propositional attitudes) and divine illumination -> mental representation (through the medieval problem of universals). 
   
   
9. What if I feel sure that two ideas are semantically related, but I can see an argument for either direction of their relative generality?
   
   If you can see strong arguments for classifying A as more general than B, and B as more general than A, and you feel that neither argument wins out, select “either/incomparable”.  As we currently use this evidence, this will prevent direct hierarchical links between the two ideas, but will still make it likely that they will be mutually reachable in the taxonomy via semantic links. 
   
   
10. What if one of the terms is ambiguous?
    
    If one (or both) of the terms being evaluated is ambiguous in philosophy, you should interpret it as having the most common meaning in the specialization under consideration.  For example, if you are asked to evaluate “internalism” in the context of epistemology, you should interpret this slightly differently than if you were evaluating it in the context of philosophy of mind (e.g., internalism about justification vs. internalism about mental content). 
    
    
11. What if I know how something is commonly classified, but I personally disagree with that classification?
    
    We are trying to capture the shared conceptualization of the discipline as it currently sees itself, not as it ought to see itself—so please try not to be radically revisionary in the feedback you provide.  However, if you know that a significant number of philosophers might go one way on a classification, but a significant number might also go the other way (e.g., it is currently recognized as a matter of dispute), feel free to “vote” in the direction you think most appropriate. 
    
    
12. What happens to my evaluations once I submit them?
    
    Your evaluations are stored in our database with a timestamp and your areas of expertise.  These evaluations are then combined with the feedback of others and encoded in statements in first-order predicate logic.  We then use nonmonotonic logic programming techniques to populates the ontology by classifying ideas in the most specific location that allows reachability to all other ideas to which it has been consistently deemed highly-related.  The result is the ontotree you can browse on our main site. 
    
    
13. What if someone else’s feedback disagrees with mine?
    
    Disagreement is currently handled in different ways, depending on whether it is disagreement in relatedness or in relative generality.  For disagreement in relatedness, we currently consider the only the highest relatedness score when populating the taxonomy.  This is based on the principle that those providing feedback are much more likely to overlook a link between two ideas than to fabricate one.  Disagreement in generality is taken more seriously.  If one expert evaluates A as more specific than B, and another evaluates B as more specific than A, the generality of the two terms is moved to incomparable/either.
    
    These measures are temporary, and will eventually be replaced with more sophisticated ways of mitigating disagreement. 
    
    
14. What does the “SEP” link do?
    
    The “SEP” link to the right of the idea pair will use the SEP’s search engine to search the SEP with a combination of the two keyword phrases.  This will produce a list of the articles that are responsible for the statistics leading to the suggestion of the candidate hyponym/hypernym pairs.
    
    
15. How should I interpret “and” and “or” categories?
    
    "A or B" should always be interpreted as the union of the two ideas (e.g., the category "ideas about materialism and dualism" would be the set composed of all the ideas about materialism plus all the ideas about dualism). "A and B" is ambiguous in the taxonomy. "And" will in many cases mean a union; in that case, the idea "A" is more specific than the idea "A and B".  In other cases, it can mean intersection, in which case the idea "A" is more general than the idea "A and B". 
    
    
16. What does “add your own” do?
    
    “Add your own” will allow you to “manually” add ideas that have not been recommended as hypernym/hyponym candidates for the idea under consideration.  The field works with an auto-complete feature that scans our database of philosophical ideas.  Currently, users cannot add new ideas that are not in our database; if our database is missing an idea you’d like to see in the ontotree, please e-mail your suggestion to inpho@indiana.edu. 
    
    
17. What do the statistics on the myspace page mean?
    
    These statistics provide various measures of the degree to which your feedback agrees with the feedback provided by other InPhO users.
    
    Rel.agree is computed by directly considering how much your relatedness judgments agree with all the feedback of others who have evaluated the same terms.  It is computed by pooling together all the feedback supplied by other experts on the terms you evaluated, and dividing the number of evaluations that agreed with yours over the total.  So, if you and two other experts evaluated A-B, C-D, and E-F, and you agreed with two other experts on each of A-B and C-D, but disagreed with both of the others on E-F, you would get a rel.agree score of 4/6 (66.7%).  
    
    Rel.std.dev works like rel.agree, but instead tells you how far your relatedness evaluations differed, on average, from those of other experts.  It is computed by taking your relatedness scores, looking at those of others who evaluated the same ideas, and taking the average difference between your judgments and those of other experts.  It is (like the relatedness evaluations themselves) on a 0-4 scale, where “0” is “in perfect agreement with everyone else” and 4 is “in perfect disagreement with everyone else.”
    
    Gen.agree is computed in the same way as rel.agree, except with generality scores.
    
    
18. Why do some categories not have any instances on the ontotree browser?
    
    Sometimes in browsing the ontotree, you will find that some most specific categories lack instances. This is likely due to the category recently being added, and not having accrued any evaluations yet. If you see a category on the ontotree without any instances, this would be an excellent place for you to provide feedback if you have the appropriate area expertise.
    
    
19. Why do some categories not produce any pair for evaluation on the idea evaluation pages?
    
    This can be caused by two things. It can mean that our methods were unable to recommend any hyponym/hypernym candidates for the candidate term, because the phrase does . This means that the category name cannot work with our automated methods and will need to be replaced; please let us know at inpho@indiana.edu. It can also mean that you have already evaluated all of the suggestions for that category. If there ideas you strongly feel should appear under that node in the tree, but have not yet been proposed, you can add them manually using the "add your own" feature.
    
    
20. Can I review/change my previous evaluations?
    
    Yes, you can review your previous evaluations at any time in the evaluation pages by clicking on the page numbers above the links. Your feedback can also be changed or deleted at any time. Simply navigate to the page where the feedback occurred (e.g. the "1", "2", "3" page links about the evaluation boxes). You can then change your evaluations for each pair by entering in new values or delete them by clicking "reset". You must then click "store evaluations for this page" at the bottom to save the revisions.
    
    
21. What if I disagree with the categorical structure of the ontotree, or feel that there are important areas being neglected?
    
    The current hierarchical structure of the taxonomy has been hand-built by editors of the SEP. We welcome suggestions for alternative classifications--please send us an e-mail at inpho@indiana.edu and share your thoughts.
    
    
22. Why hasn't my feedback added any new instances to the ontotree?
    
    The nonmonotonic logic programming step takes some time, and cannot be run every time a user inputs new feedback. Instead, the program is run every evening; check back in a few days to see how your feedback has influenced the structure of the taxonomy.
    
    
23. How hard should I think about idea pairs which seem odd?
    
    Our statistical methods are imperfect, and, like any automated methods, will generate a small amount of "junk". If you are curious why two ideas have been paired, you are encouraged to investigate using the "SEP Google Search" button in the pair's evaluation box; you may find a relationship of which you were unaware. However, if you cannot determine any relationship between the two, don't waste too much time thinking about it -- just skip it and move on to the next pair.