How do you evaluate someone's knowledge? What are the costs of
doing so? How can we maximize cost/benefit ratios based on our concern
In determining whether or not someone should gain access, we want
to be able to evaluate their knowledge. Yet, the cost of evaluation
is directly correlated to the accuracy of the test. Depending on
the level of accuracy necessary, one must balance between cost of
evaluation and acceptable level of potential deception.
In evaluating real world traits, we use one of two types of signals:
assessment or conventional signals. Assessment signals are
challenging to maintain, but they are pretty accurate markers of
the underlying trait (i.e. big muscles = strength). Alternatively,
conventional signals are easy to maintain but also easy to
falsify (i.e. wearing a powerlifting t-shirt). Likewise, the cost
in evaluating signals can vary. For example, while someone may look
like a skateboarder, asking them to prove themselves by showing
tricks is costly.
In the digital realm, one can simply state an email address, but
proving that it really is them is costly. This is even more tricky
for evaluating identity information - how do you prove that one's
skateboarding website truly marks them as a skateboarder? Besides,
depending on the context, someone's interest in skateboarding may
be more valuable than one's actual ability. For this reason, SecureId
gives you multiple options for evaluating one's knowledge.
Question/answer. A question/answer system is the most obvious
way to evaluate someone's knowledge. By asking someone a question,
you are ignoring any potential signals and requiring them to prove
themselves. Yet, such evaluation is expensive - it's timely and
a hassle to manage and maintain. You must determine the proper question
and type of potential response so that it's only answerable by the
category of people that you want to include.
Open Q&A. An open Q&A system is a fairly guaranteed
way to assess someone's knowledge. Without hints, the individual
must know the information and guessing is tremendously difficult.
Unfortunately, such a system is also the most problematic to build
and maintain. For example, imagine that the question is "Who
is my lover?" and the creator inputs "Bob" but
the respondent inputs "Robert." Although we'd probably
assume that this is accurate, the computer system might not.
Limited Q&A. By giving a user a list of potential
answers, you eliminate the problem of improperly formatted responses,
but you also make it much easier to guess the potential answer.
Thus, the evaluation is no longer as strong and wording the question
so that it can't easily be guessed is quite tricky.
Assessment signals. One way of evaluating people is based
on the information that they have in their system that cannot be
faked. For example, assuming that SecureId requires you to
confirm your email address, you should be able to assess others
based on their email address. As such, the system could determine
that anyone with an @mit.edu can be assumed to be associated with
Conventional signals. In order to determine someone's knowledge,
you could also use unconfirmed data within one's profile. For example,
if someone lists skateboarding as an activity that interests them
and you are allowing anyone interested in skateboarding to access
a certain context, the system could evaluate this without requiring
proof. Of course, this could mean that individuals who are not skateboarders
have access simply because they state an interest.
Since different evaluations have different costs for evaluation
and management, it's important to determine what level of deception
is acceptable. The more secure you want certain information to be,
the more you must spend on evaluating the individual's knowledge.