May 2020 – Open ARDF

Ranking Part 3: Ranking Criteria

Part one of this series of blog entries presented one reasonable approach to scoring ARDF competitions. Part two described a way to adjust competitor scores to make them apple-to-apple comparable from one event to the next. Part three examines how the adjusted scores might be brought together to provide a meaningful ranking system for USA ARDF athletes.

The goal of creating a ranking system is to display competitors’ names by order of their overall ARDF performance over some period of time. That is, their names should be ordered 1st, 2nd, 3rd, etc. based on how well they performed in recent ARDF competitions. Certain standards and criteria need to be applied in order to determine who gets ranked, and how their rank is determined. This post examines what some of those standards and criteria might be. It is not meant to impose any particular set of standards, but rather to present one approach that can serve as a starting point for discussion.

A competitor’s rank is just one criterion that might be used when selecting team members. Being highly ranked does not guarantee that a competitor will receive an invitation to join the team. Conversely, not being listed among the rankings does not mean that an individual won’t be selected for team membership. The team selection subcommittee is free to utilize all reasonable criteria to ensure that the team selection process is fair and equitable. Rankings are just one tool they may use to assist with the team selection process.

Who To Rank

A ranking system should not attempt to include every individual who has ever played the sport. It is fitting that only those who have demonstrated a certain mastery of the sport be included in the rankings. So the proposed system selects those individuals who meet the basic requirements.

Since the rankings are for USA competitors, it seems fitting that only those who might qualify for USA team membership be included. The USA ARDF Rules currently describe the basic requirements as follows:

Competitors eligible for USA ARDF Champion shall be citizens of the USA, Green Card holders, or have lived in the USA for the previous year and shall not have competed for the title of Champion in any other country.
USA RULES FOR AMATEUR RADIO DIRECTION FINDING Version 9-Feb-2020

Recent competitive ARDF experience is also an important criterion. Having met the basic requirements for team membership, ranked competitors also need to have competed in at least one recent ARRL-sanctioned ARDF event, and to have demonstrated a minimum level of mastery of the sport.

A suggested definition for a recent competition is an ARRL-sanctioned ARDF competition held during the two most recent calendar years in which ARRL-sanctioned events were held. Usually, that will be competitions held during the current year and the previous year. But COVID-19 demonstrated that there might be years in which no ARRL-sanctioned ARDF events are held.

As a minimum level of ARDF mastery, the suggested measure is this: in at least one recent sanctioned competition the competitor must have successfully located all assigned transmitters and registered at the finish before the time limit expired. In other words, they successfully completed at least one course. One might ask, “Why not rank everyone who didn’t OT?” There are two main reasons why assigning a rank to an individual should require a higher minimum standard: 1) It gives greater significance to the achievement of attaining ranked status, and 2) It may not be feasible to calculate an accurate rank for all who participated.

Expanding on the second point: consider that in a 5-transmitter M21 Classic competition a 5-fox minimum requirement results in just one possible combination of foxes and only one permutation that must be selected as the optimum order. But for those who find only four out of five foxes, there are five different combinations of foxes that must be analyzed to identify the one permutation that is the optimum order. If three out of five foxes are found then there are ten different fox combinations and ten more optimum permutations that must be analyzed. For two out of five foxes there are again ten combinations and permutations to be analyzed. And if only one fox is found there are five combinations. That means that there is potentially 5 + 10 + 10 + 5 = 30 times as much ranking analysis required if “not going OT” is used as a minimum ranking requirement in a five-fox competition. And that is just for the M21 category. When there are few competitors this will be a manageable problem, but clearly, setting too low a bar does not allow the ranking system to scale.

Remember that being ranked is not a prerequisite for being selected for Team USA. So not being ranked need not exclude any qualified individual from making the team. The Team Selection Subcommittee has latitude to take into account other factors, including extenuating circumstances when extending invitations for team membership. So, as a practical matter, restricting the number of those ranked, and having the team-selection process consider unranked individuals using a non-analytical methodology is the most reasonable approach.

Calculating Rankings

A competitor who meets all the criteria described above will have their rank calculated and listed in the rankings tables. Separate rankings will be calculated for each competition format: Classic 80m, Classic 2m, Sprint, and Foxoring. Within each format rankings table, the competitors will be ranked separately by age/gender category.

All qualifying competitors are eligible to receive up to four separate rankings: one for each competition format. Those rankings will be made relative to all other qualifying competitors in their age/gender category.

A competitor’s adjusted competition scores (see Part 2) will be used for determining rank. Only the highest event score achieved during a calendar year will be considered. (This will ensure there is no disincentive to participate in multiple events annually.) The highest score from both recent years will be averaged to determine a competitor’s ranking score. If the competitor only participated during one recent calendar year, then that year’s highest score will be the competitor’s ranking score. A competitor with a higher ranking score will be ranked above any competitor with a lower ranking score. Tied competitors will both receive the same rank, with the next lower rank(s) left vacant.

If a competitor lacks results from recent competitions because he/she helped conduct a sanctioned event, then their rankings will be calculated using the two most recent years for which they have qualifying results, going back up to four years.

Discussion

No ranking system is perfect. The one described above is no exception. But it does have simplicity working in its favor. The formulae it uses are simple, provably stable, and give predictable results. The entire system can be written in a short Javascript program and rendered on any browser. The results, and the source code, can be shared freely so that competitors and organizers can analyze it and improve upon it.

Ranking Part 2: Adjusting for Difficulty

ARDF is conducted in nature. So, like golf, mountain bike racing, or skiing, the venue influences the game’s difficulty level. Navigating in a sparse forest is easier than in jungle-like undergrowth. Running on flat smooth terrain doesn’t require as much effort as climbing steep rocky hills. Some ARDF courses are going to be more difficult than others, and that will always be the case.

So when a competitor scores highly on a difficult course it says more about his/her fitness and skills than acing an easier course. And because an ARDF score is derived from one’s finish time relative to other competitors on the course, prevailing against stiff competition is also a better indicator of an athlete’s skills and fitness level.

Venues across the USA have diverse terrains, and course difficulty can vary widely. So if we are going to compare scores achieved in competitions held at different venues those scores need to be adjusted accordingly. Ideally, the scoring system would be designed to handle adjustments automatically. In orienteering this is accomplished by calculating scores relative to the average times of the top finishers. Because orienteering competitions consistently have a sufficient number of highly-skilled entrants, the top performances establish a good baseline by which to compare the subordinate finishers.

But many ARDF categories at USA events lack highly-competitive entrants and sometimes contain only a single competitor. So the question arises: how to adjust ARDF scores to make them comparable from one event to the next?

Ideal Time

One proposal is to utilize the concept of ideal time ( $T_{i}$ ). Ideal time is the expected finish time for an elite ARDF athlete traversing a course with the following results:

Finds each assigned transmitter in optimum order following the shortest practical route based on the competition map;
Unaffected by chance. E.g., doesn’t find any flags off-cycle, makes no wrong turns, etc.;
Runs at a speed that is realistic for his/her age and gender category taking into account the running conditions present along the route followed.

Note that ideal time is not YOUR ideal time, nor is it the ideal time of any of the competitors in the competition. Ideal time represents what a healthy elite athlete should be able to accomplish without anything unpredictable happening.

There isn’t currently a software program capable of calculating ideal time, so it is a manual process. Although it is a manual process, it is not very difficult, and the process can be documented in detail in order to allow almost anyone to perform the calculation. Calculating ideal time involves spending some time analyzing the competition map, plotting the shortest practical route from start to finish along the optimum order of transmitters, and estimating a realistic speed of movement along that route. A unique ideal time is assigned for each course for each competition held during an ARDF event.

Once calculated, the ideal times represent the results that an elite athlete in each age/gender category could have achieved. Ideal time can be applied to the scoring approach presented in Part 1. By doing so, the scores of all competitors who found all the assigned transmitters will be adjusted, making them more reflective of what they would have scored had an elite athlete actually participated.

Adjusting for Competition

Calculating ideal time and inserting it into the race results has the effect of adjusting all competitors’ scores to reflect what they would have been had an elite competitor actually participated. This will be the case even for categories in which only a single competitor participated. Utilizing ideal time in this manner doesn’t require any special formulas, and the adjusted scores still reflect the correct finish order.

Examples

Consider the M40 results from a Classic 80m competition in which the ideal time was calculated to be 60 minutes (3600 seconds)and there were five transmitters for the M40 competitors to find.

(Example 1) M40 competitor “A” found all five transmitters and finished with a time of 75 minutes (4500 seconds). He had the best overall time (i.e., $T_{1} = T_{A}$ ) amongst all M40 competitors who found all five transmitters. Without putting ideal time into the results, competitor “A” would have a score of:

$S_{A} = (5 + T_{1}/T_{1})\cdot 16.7 = 6 \cdot 16.\overline6 = 100.0$ points

Including ideal time ( $T_{i}$ ) in the results adjusts the score of competitor “A” as follows:

$S_{A}^i = (5 + T_{i}/T_{A})\cdot 16.7 = 5.8 \cdot 16.\overline6 = 96.\overline6$ points

(Example 2) M40 competitor “B” who found only four of the five assigned transmitters. No adjustment for ideal time would be calculated for this competitor. Ideal time is only calculated for a course that includes all the assigned transmitters. Those who find fewer transmitters will not receive adjusted scores.

(Example 3) Let’s look at a very slow M40 competitor “C” who found all five transmitters but required the full time limit of 3 hours (10800 seconds). His unadjusted score would be:

$S_{C} = (5 + T_{1}/T_{C})\cdot 16.7 = 5.41\overline6 \cdot 16.\overline6 = 90.2\overbar7$ points

Adjusting for ideal time gives:

$S_{C}^i = (5 + T_{i}/T_{C})\cdot 16.7 = 5.\overline3 \cdot 16.\overline6 = 88.\overline8$ points

Adjusting for Course Difficulty

The value calculated for ideal time will be greater for difficult courses and lesser for easier courses. Therefore, by comparing the ideal time calculated for an actual course with the ideal time calculated for a standard reference course, a unitless ratio can be derived. That ratio can be used to adjust competitor scores to account for the course’s difficulty.

So long as the standard reference is consistent the adjustment will be fair, since it will affect all competitors’ scores identically. So, for both consistency and simplicity, a reference of one hour is suggested for Classic and Foxoring courses, and fifteen minutes for Sprint, since those are the generally-used length targets used by most course designers. This reference can be named the ideal reference time $(T_{i}^R)$ and uses the same time units (seconds) as ideal time.

The unitless ratio to be used for adjusting for course difficulty is calculated as follows:

(1) $\begin{equation*}\gamma = T_{i} / T_{i}^R\end{equation*}$

Where:
$\gamma$ is the unitless course difficulty factor,
$T_{i}^R$ is the ideal reference time, and
$T_{i}$ is the ideal time.

The more difficult a course is the greater its ideal time. If the course designer made a course so that elite competitors complete it in the targeted amount of time (1 hour Classic/Foxoring or 15 minutes Sprint) then $T_{i}^R \approx T_{i}$ so that $\gamma \approx 1$ . And as a course is made more difficult the value of $\gamma$ increases above $1$ , and less difficult courses will have $\gamma$ values less than $1$ .

$\gamma$ only applies to those factors affecting a competitor’s overall time. Therefore, only the time component of a competitor’s score should be multiplied by $\gamma$ . $\gamma$ should not be applied to the component of a score derived from the number of transmitters found.

When $\gamma$ is multiplied by the time component of a competitor’s course score adjusted for the ideal time, the result is the adjusted score ( $S_{c}^{adj}$ ) that can be compared apples-to-apples to any other adjusted score for the same type of competition and age/gender category. Since transmitter count is not involved in the computation of the adjusted score, it is fine to rescale the score without regard to the number of transmitters. So the formula for adjusted score is simply:

(2) $\begin{equation*}S_{c}^{adj} = 100 \cdot \gamma \cdot T_{i} / T_{c}\end{equation*}$

Where:
$S_{c}^{adj}$ is the adjusted score for competitor $c$ ,
$\gamma$ is the course difficulty factor,
$T_{i}$ is the ideal time for the course, and
$T_{c}$ is the finish time for competitor $c$ .

Inserting the right side of Equation 1 for $\gamma$ into Equation 2 gives the following equation for $S_{c}^{adj}$ that allows us to skip the calculation of $\gamma$ :

(3) $\begin{equation*}S_{c}^{adj} = 100 \cdot T_{i}^2 / (T_{i}^R \cdot T_{c})\end{equation*}$

Where:
$S_{c}^{adj}$ is the adjusted score for competitor $c$ ,
$T_{i}$ is the ideal time for the course,
$T_{i}^R$ is the ideal reference time,
$T_{c}$ is the finish time for competitor $c$ .

Example

(Example 4) Consider the M40 results from a Classic 80m competition in which the ideal time ( $T_{i}$ ) was calculated to be 60 minutes (3600 seconds).

With the information above, calculate the adjusted score ( $S_{c}^{adj}$ ) for Example 1. First, select the appropriate ideal reference time:

$T_{i}^R = 3600$ seconds (For a Classic competition.)

Use $T_{i}^R$ , $T_{i}$ , and $T_{c}$ to calculate the adjusted score for competitor “A” from Example 1. Using Equation 3:

$S_{A}^{adj} = 100 \cdot T_{i}^2/(T_{i}^R \cdot T_{A})$

$= 100 \cdot (3600)^2 / (3600 \cdot 4500) =$ 80.0 points

Other Comments

Because ideal time assumes no chance events, it is possible that some elite competitors will achieve finish times shorter than ideal time. When that happens in the scoring system described in Part 1, those athletes will receive scores greater than 100.0 after the ideal time is applied to the overall results. Such outcomes are acceptable and should be interpreted as indicating that some transmitters were probably found off-cycle.

The scoring methodology described in these posts does not attempt to adjust for differences between the various competition formats, competitor age, or gender. Therefore, adjusted scores must only be compared or combined with other adjusted scores for the same competition format and age/gender category.

There have been some questions regarding Equation 3 and its use of the square of ideal time ( $T_{i}^2$ ). The ideal time is squared in the calculation of adjusted score ( $S_{A}^{adj}$ ) because it is being used to compensate for two separate and independent factors: the strength of the competition, and course difficulty. So it is necessary for it to be applied twice.

Note that the adjusted score contains only a time component and therefore isn’t affected by transmitter count. The effects of transmitter count were accounted for in the calculation of ideal time and adjusted score. So it is reasonable to combine (e.g., average) and compare adjusted scores within the same age/gender category from competitions of the same format having differing numbers of transmitters.

Experimentation with the formulae above is encouraged. Their use over time will no doubt reveal improvements to this scoring approach.

Summary

Because ideal time assumes the technical skills and fitness level of an elite athlete, it can be used to adjust for the absence of strong competition at a particular event. Because ideal time takes into account the actual course lengths and terrain, it can be used to adjust for the effects of the venue on competitor scores. Because the process of calculating ideal time can be documented and repeated by trained individuals, it can be applied objectively and precisely.

But, ideal time is not an ideal solution. It would be better to have a sufficient number of elite competitors at each ARDF competition to utilize orienteering’s scoring approach. But until the day of highly-competitive USA ARDF contests arrives, the use of ideal time as presented above is a reasonable approach allowing results to be compared between events having different levels of difficulty, even when few (or no) elite competitors were in attendance.

The approach described in this writing will not impact the order of finish for any event, but it achieves two important goals:

It allows competitors to see how they would have scored relative to elite competitors like those they might find at a World Championships.
It provides a way to fairly compare scores from events held at different venues and presenting different levels of difficulty.

Number 2 above will be examined more closely in Part 3.

Ranking Part 1: Scores for ARDF

Among other duties, the ARRL ARDF Committee is tasked with establishing a fair and equitable ranking system for use in team selection. It has been rightly pointed out that there are many considerations when selecting a team. Some considerations deal with subjective factors that have little or nothing to do with how well a candidate performed historically at ARDF events.

But competitive performance is one useful and objective factor that can be readily quantified and analyzed. And so competitive performance at ARRL-sanctioned ARDF events should be calculated and utilized as one component of a fair team selection process. It will also be helpful to competitors wanting to track their performance relative to their competition.

ARDF rules stipulate that the place (1st, 2nd, 3rd, etc.) of an individual competitor depends firstly on the number of transmitters found, and secondly on his or her running time. The rules provide no numerical formulae for deriving a single number to represent a competitor’s performance. But such a number would be helpful for facilitating a mathematically-rigorous ranking system, and would be the first of three steps to establishing such a system:

Establish a scoring system for ARDF competitions.
Define an approach for adjusting competition scores to make them comparable between courses with different difficulty levels.
Create a formula for combining adjusted scores to obtain a competitor’s rank relative to other competitors in his or her age/gender category.

The subject at hand is the first bullet above: how to represent a competitor’s ARDF performance at a single competition as a single numerical score. The other two bullets will be addressed in later postings.

Scores for ARDF

ARDF rules require that both the number of transmitters found, and the amount of time a competitor takes to reach the finish line, be used to determine a competitor’s place. And since a competitor’s score should accurately reflect their placement relative to other competitors, the score calculation must also utilize both the number of transmitters found, and a competitor’s total time, to derive their score.

Since the rules require the total transmitters found to be decisive in a competitor’s placement, time must never contribute more to a competitor’s score than does a single transmitter. In other words, no matter how fast a competitor runs a course, his/her score for that race must be less than the score of any competitor who found more transmitters. The rules demand it.

Most other properties of a score may be chosen arbitrarily. For instance, whether a score is proportional or inversely proportional to a competitor’s placement is totally a matter of convention. So, to keep things simple, the following scoring conventions will be used here:

A higher score reflects a higher (better) placement with a score of 100.0 being awarded to the first-place finisher(s) of an age/gender category.
All transmitters count the same amount toward the calculation of a competitor’s score.
Competitors’ times are scored relative to the fastest time of any competitor who found the same number of transmitters.

Item 3 is worthy of additional discussion. Note that all competitors within an age/gender category are always assigned the exact same course to run, and therefore the same number of transmitters to find. But the rules allow any competitor to choose, at their own discretion, to find only a subset of the assigned transmitters. A competitor might elect to find fewer transmitters if, for instance, finding more transmitters seems likely to result in their being classified Overtime (OVT). By choosing to find fewer than the assigned number of transmitters a competitor can avoid being classified OVT. But by choosing to find fewer transmitters a competitor accepts a placement below all the competitors who find more of the assigned transmitters than they. So, in effect, by choosing to find fewer transmitters they place themselves in a separate category of finisher. Since the rules effectively segregate finishers’ placements by the number of transmitters found, it is appropriate then that their times be likewise segregated and compared only to the times of others in their category who elected to find the same number of transmitters.

Score Calculations

Since all transmitters count the same toward a competitor’s score, and a competitor’s speed performance (time) must count no more than a single transmitter, each transmitter and the time component of the score is assigned a maximum score value of:

(1) $\begin{equation*}s_{N} = 100 / (N + 1)\end{equation*}$

Where
$s_{N}$ is the point value of each transmitter found (and maximum points awarded based on finish time),
$N$ is the total number of transmitters assigned to a competitor.

So, for example, if an M40 competitor is assigned five transmitters to locate, then each of the transmitters will have a score value of $100 / (5 + 1)$ or $16.\overline6$ points. The maximum number of points the M40 competitor can gain by having the fastest overall time among all M40 competitors on that course is also $16.\overline6$ points. So the total score for competitor “c” is given by:

(2) $\begin{equation*}S_{c} = (n_{c} + T_{1}/T_{c})\cdot s_{N}\end{equation*}$

Where
$S_{c}$ is the score for competitor $c$ ,
$n_{c}$ is the total number of transmitters found by competitor $c$ ,
$T_{1}$ is the overall time in seconds of the fastest of the competitors who found $n$ transmitters,
$T_{c}$ is the competitor’s overall time in seconds, and
$s_{N}$ is the value derived in equation (1)

Examples:

(Example 1) M40 competitor “A” is assigned to locate five transmitters and finds all five, and also has the best overall time (i.e., $T_{1} = T_{A}$ ) amongst all M40 competitors who found all five transmitters. He would have a score of:

$S_{A} = (5 + T_{1}/T_{1})\cdot 16.7 = 6 \cdot 16.\overline6 = 100.0$ points

(Example 2) M40 competitor “B” who found only four transmitters, and who had the best overall time amongst all M40 competitors who found four of the transmitters would have a score of:

$S_{B} = (4 + T_{1}/T_{1})\cdot 16.7 = 5 \cdot 16.\overline6 = 83.\bar3$ points

(Example 3) Let’s look at a very slow M40 competitor “C” who found all five transmitters but required 10 times as long to reach the finish line as did the competitor in Example 1:

$S_{C} = (5 + T_{1}/(10 \cdot T_{1}))\cdot 16.7 = 5.1 \cdot 16.\overline6 = 85$ points

Note in Example 3 that the slow finisher finding all five transmitters still received a score substantially higher than the best competitor who found only four transmitters in Example 2.

Other Observations

Note that equation (1) implies that the more transmitters assigned to competitors the smaller the effect of each individual transmitter and of time on the competitors’ scores. Consider: if there were only one transmitter to find, equation (1) says that finding the single transmitter would account for 50 points, and one’s finish time could account for up to an additional 50 points. But if we increase the number of transmitters to nine, then finding a single transmitter adds just 10 points to one’s score and time accounts for just 10 points as well. This can be difficult to reconcile when combining scores (e.g., for averaging, etc.) from competitions in which different numbers of transmitters were assigned.

To avoid the pitfalls of mixing or comparing scores derived from differing time and transmitter component values, the following approach is suggested:

Only scores for the same competition format will be combined or compared.
Only scores for competitors finding the same number of transmitters on the same course in the same age/gender category will be combined or compared.

Limitations of Scoring

The above approach for deriving a single score for competitors’ ARDF race results can be helpful for comparing performance among those competing within the same age/gender category and on the same course. But it would fall short if it were used to compare performances on courses of significantly different levels of difficulty. That subject will be addressed in Part 2.