How do you evaluate someone's knowledge? What are the costs of doing so? How can we maximize cost/benefit ratios based on our concern for deception?

In determining whether or not someone should gain access, we want to be able to evaluate their knowledge. Yet, the cost of evaluation is directly correlated to the accuracy of the test. Depending on the level of accuracy necessary, one must balance between cost of evaluation and acceptable level of potential deception. In evaluating real world traits, we use one of two types of signals: assessment or conventional signals. Assessment signals are challenging to maintain, but they are pretty accurate markers of the underlying trait (i.e. big muscles = strength). Alternatively, conventional signals are easy to maintain but also easy to falsify (i.e. wearing a powerlifting t-shirt). Likewise, the cost in evaluating signals can vary. For example, while someone may look like a skateboarder, asking them to prove themselves by showing tricks is costly.

In the digital realm, one can simply state an email address, but proving that it really is them is costly. This is even more tricky for evaluating identity information - how do you prove that one's skateboarding website truly marks them as a skateboarder? Besides, depending on the context, someone's interest in skateboarding may be more valuable than one's actual ability. For this reason, SecureId gives you multiple options for evaluating one's knowledge.

Question/answer. A question/answer system is the most obvious way to evaluate someone's knowledge. By asking someone a question, you are ignoring any potential signals and requiring them to prove themselves. Yet, such evaluation is expensive - it's timely and a hassle to manage and maintain. You must determine the proper question and type of potential response so that it's only answerable by the category of people that you want to include.

Open Q&A. An open Q&A system is a fairly guaranteed way to assess someone's knowledge. Without hints, the individual must know the information and guessing is tremendously difficult. Unfortunately, such a system is also the most problematic to build and maintain. For example, imagine that the question is "Who is my lover?" and the creator inputs "Bob" but the respondent inputs "Robert." Although we'd probably assume that this is accurate, the computer system might not.

Limited Q&A. By giving a user a list of potential answers, you eliminate the problem of improperly formatted responses, but you also make it much easier to guess the potential answer. Thus, the evaluation is no longer as strong and wording the question so that it can't easily be guessed is quite tricky.

Assessment signals. One way of evaluating people is based on the information that they have in their system that cannot be faked. For example, assuming that SecureId requires you to confirm your email address, you should be able to assess others based on their email address. As such, the system could determine that anyone with an @mit.edu can be assumed to be associated with MIT.

Conventional signals. In order to determine someone's knowledge, you could also use unconfirmed data within one's profile. For example, if someone lists skateboarding as an activity that interests them and you are allowing anyone interested in skateboarding to access a certain context, the system could evaluate this without requiring proof. Of course, this could mean that individuals who are not skateboarders have access simply because they state an interest.

Since different evaluations have different costs for evaluation and management, it's important to determine what level of deception is acceptable. The more secure you want certain information to be, the more you must spend on evaluating the individual's knowledge.