Programming questions are created in the same way as other question types, but a specific tool is available to facilitate the creation of this type of question and to place specific values for programming questions.
To access this tool, go to the “Tools” menu and select “Programming Question Builder”. After loading the question, the corresponding information will be displayed.
On the right side, there is an editor where you can test how the question would evaluate an answer, and, on the left side, there are the conditions (validation scenarios) to validate the question, which is composed of two sections; the first one that is used to validate the question when the candidate is answering and the ones that will be used to validate and determine the score of the question. This difference exists to prevent the candidate from entering a trivial answer such as “Print 5” instead of an algorithm to solve the question problem. This is why it is usual to include predefined code, one to validate during the exam and another to assign the score (with other values) so that the candidate cannot enter a trivial answer without the corresponding logic.
For both validation during the test and for scoring, you need to enter the expected value, which you can enter directly, or you can write a program to test the question and then copy the output from it to the answers. You can run your program with any of the conditions by clicking the corresponding Run button. The conditions to validate weight since it is possible to place several different conditions, each with a different weight that indicates how much weight this condition has in calculating the score obtained by the candidate in the question.
Optionally, you can set up a code efficiency evaluation, based on the number of instructions (tokens) used by the program. This is to give less score to solutions that are longer than necessary. For this, you must fill in the Min Token, Max Token, and Code Efficiency Weight fields. Code Efficiency Weight refers to what percentage of the question score will depend on the coding efficiency. For example, if the question has a score of 20 and a weight of 50, then 50% of the score (10) will depend on the efficiency. The code is evaluated and, if the number of tokens is less than or equal to the Min token, it will get the maximum score for efficiency. On the other hand, if the number is greater than the Max token, it will receive 0. It is advisable to configure this only when you already have a sample of several candidate responses to identify reasonable ranges of the Min and the Max Token (when executing the candidate response in the constructor, the Min and Max token values for the executed code are shown).
Finally, for questions that support measuring code performance, it is possible to indicate the ranges and the performance weight in the question score. It is necessary to mark which validation scenarios will be used to measure performance. Currently, only questions that use the TLang module allow performance to be measured.