Test ROODT rapport

Roodt Rapport

The questions of validity and reliability are often raised with regard to Discus and other similar systems. This section of the site reproduces a South African study, A Reliability and Validity Study on the Discus Personality Profiling System, by Karin Roodt of the prestigious Technikon Natal. Note that this study - known as The Roodt Report - relates not just to DISC in general but specifically to the Discus software.

A Reliability and Validity Study on the Discus Personality Profiling System

Abstract

The purpose of the investigation was to determine whether the Discus measuring instrument could be considered a reliable and valid instrument. The test-retest method was used in the reliability study and was administered to 90 employees from a variety of companies in Kwa Zulu-Natal and Gauteng. The Pearson's product-moment correlation coefficient was used and correlation scores of 0.728 (Dominance), 0.645 (Influence), 0.730 (Steadiness) and 0.550 (Compliance) were established. The p-value in all the cases was as low as 0.0001. This indicates significance at alpha = 0.001. It can therefore be concluded with 99.9% level of confidence that the Discus instrument is reliable.

In the validity exercise criterion-related validity was used. An exploratory study was undertaken in order to determine which of the 15 Factors (Factor B excluded) of the 16-PF correlated with the four dimensions of the Discus. One hundred and twenty respondents in South Africa were involved for this purpose. The Pearson's product-moment correlation coefficient was applied. It was found that Factors Q1, X=G, L, Q1 and X=Q2, E; E, Q2 and -I show significant correlations with Dominance at the 1% and 5% level of significance. Factors A, -Q2, H, F and -Q3 show significant correlations with Influence at the 1% and 5% level of significance. Factors -E and -Q1 show a correlation with Steadiness at the 5% level of significance. Factors -E, Q2, -H, -G and O show significant correlations with Compliance at the 1% and 5% level of significance. It can therefore be concluded that the correlations were significant.

What follows is the outcome of a reliability and validity study as performed by Karin Roodt, M Ed (Counselling and Guidance) UNISA, Registered Psychologist (SAMDC), Senior Lecturer, Technikon Natal (serving as Project Manager), assisted by Charles Robert, BSc (Hons)(Stats): Statistician, Director of South Africa on Line.

I would like to thank the following companies and individuals for participating in this study and in some way contributing to the final report:

Labour Lawyer
Graham Giles for scrutinising this report from a law perspective

A study of this nature must be approached with caution because no single psychological tool is able to yield everything about a person's personality. A test battery, i.e. a collection of several tests, reveals far more about an individual than does any single test of personality, preference, interests, intelligence or general personality dynamics.

K. Roodt 31.01.1997

"The Discus personality profiler is a completely computerised assessment tool, designed to describe the different roles a person fulfils in the work environment."

(Axiom 1994:1)

The word 'personality' has been debated for centuries. Everybody has their own idea about exactly what it means. In Discus terms a personality is defined as the sum of all a person's varying response styles to varying stimuli (Swanepoel 1995:1). In practical terms, however, it is impossible to measure and evaluate every one of a person's possible responses to every possible stimulus.

The purpose of this study is to determine whether the Discus instrument may be considered a reliable and valid instrument for assessing personality.

3.1 Reliability

In the reliability exercise, the test-retest reliability technique was used. According to this method the same instrument is applied to the same respondents at a later stage and the correlation between the two scores is then calculated (Huysamen 1980:54; Mulder 1981:211).

The questionnaire was administered by the respective people participating in the exercise. All of these participants are trained in Discus and how to administer the instrument. The instrument was administered for the first exercise to obtain a pretest score. The exercise was then repeated with the same respondents after a period of three months in order to obtain a post-test score.

A statistical evaluation of the raw data, resulting from the exercise, was then obtained by using the SAS system, reflecting Pearson's Product-moment correlation coefficient (coefficiency of stability).

3.1.1 The Questionnaire

The questionnaire consists of 24 questions each of which presents the respondents with four options. The respondents' task is to select one of the options that most closely resembles themselves, and one that least closely describes them. The respondents are required to focus on the role they fulfil in their work environment and answer all the questions in relation to that role.

Phrase-based
The phrase-based question set contains questions of the form 'Behaving compassionately towards others' or 'Persuading others to your point of view'.

Adjective-based
The adjective-based question set contains words such as 'kind-hearted', 'persuasive' and 'modest'.

For the purpose of this exercise the phrase-based questionnaire was used because it is easier to understand.

3.1.2 Sampling Technique and Size

Various companies were approached to assist with the exercise as reflected in table 1.

The questionnaire was administered to 90 respondents. These respondents were randomly selected from the respective companies reflected in table 1. A statistical evaluation of the raw data resulting from the testing was then obtained by using the SAS system reflecting Pearson's product-moment correlation coefficient.

The questionnaire was administered to 90 respondents. These respondents were randomly selected from the respective companies reflected in table 1. A statistical evaluation of the raw data resulting from the testing was then obtained by using the SAS system reflecting Pearson's product-moment correlation coefficient.

The significance level chosen for this instrument is alpha = 5%. Where the p-value is less than 0.05, the scores show a significant correlation. In the reliability analysis the p-value in all the cases is as low as 0.0001. This indicates significance at alpha = 0.001. It can therefore be said the correlation is significant at 1% level.

The reliability coefficient of the measuring instrument is close to 1 and can therefore be seen as reliable.

3.2 Validity

3.2.1 Content Validity

Content validity of the instrument is determined when the instrument is designed. Content validity refers to the extent in which the instrument measures what it is supposed to measure (De Wet, De K Monteith, Steyn& Venter 1981:146; Huysamen 1980:95; Mulder 1989:219).

Each question in the Discus instrument was evaluated by the designers of the instrument, namely Axiom. Although content validity was done by Axiom, the researcher also decided to measure validity in terms of criterion-related validity.

3.2.2 Criterion-related Validity

By applying the method of criterion-related validity, an exploratory study was done by correlating all 15 Factors in Cattell's 16-PF with the four dimensions in the Discus. Factor B was not considered as doubts exist in the literature as to the validity of Factor B (intelligence) within a personality test.

Criterion-related validity was restricted to validation procedures in which the test scores of a group of respondents are compared with ratings of other measurements (Aiken 1994:96).

Nunally (1978) claims that it is unrealistic to expect exceptionally high correlation coefficients and Anastasi (1976) says that coefficients of 0.20 and higher can be significant.

3.2.3 Sampling Size and Technique

In this exercise employees of the Edgars group, Toyota South Africa and Technikon Natal were used. It was therefore decided to use the Discus and Cattell's 16-PF (Form A) for this exercise.

In an attempt to determine a correlation between the Discus dimensions and Cattell's 16-PF, scores on the 16-PF were obtained from 120 employees employed by the abovementioned companies. These respondents were randomly selected from line managers, middle managers, professionals and junior officials. The sample was drawn from all organisational functions and cultural groups within the organisations.

3.2.4 Measuring Instruments

3.2.4.1 The Discus Questionnaire

This questionnaire has already been discussed in paragraph 3.1.1.

3.2.4.2 Cattell's 16-PF

The 16-PF is specifically constructed for the purpose of determining individual attitudes, perceptions and personality characteristics. It was developed by R.B. Cattell and published in 1949. The A and B Forms of the test consist of 187 items each and are suitable for adults with at least standard 10 or equivalent education. The 16-PF can be used for the evaluation of personality in people of different population groups because it is culture friendly (Prinsloo 1992:21-22).

Cattell applied the technique of factor analysis and obtained a set of 16 primary factors. The rationale behind the 16-PF is that a questionnaire which is based on revealed traits, obtained through mathematical techniques from a large pool of possible personality descriptions, is capable of measuring reliably and validly the true constructs present in humans.

The general purpose of the 16-PF is to describe testees' personality and predict behaviour using a set of selected, structured items. The test has many practical applications, some of which are mentioned below:

Careercounselling

Fruitful employment in industrial and career settings

Counselling

Clinical settings

Extremelyusefulresearch

The results of a factor analysis completed by Cattell identified the following 16 factors (table 3).

TABLE 3: THE HIGH AND LOW FACTORS OF THE 16-PF

LOW SCORES

FACTOR
1 2 3 4 5 6 7 8 9 10

HIGH SCORES

Reserved

A

Outgoing

Dull

B

Bright

EasilyUpset

C

Calm

Submissive

E

Dominant

Serious

F

Impulsive

Frivolous

G

Responsible

Shy

H

Bold

Tough-minded

I

Tender-minded

Trusting

L

Suspecting

Practical

M

Fanciful

Forthright

N

Calculating

Secure

O

Apprehensive

Conservative

Q1

Liberal

Group-dependent

Q2

Self-sufficient

Uncontrolled

Q3

Controlled

Relaxed

Q4

Tense

For interpretation purposes, factor scores of 1 to 3 and 8 to 10 are considered. The low numbers of each factor are pictured as portraying one extreme of the profile and the high numbers as portraying the other. It should be pointed out that Cattell, in analysing all 16 factors, came up with clusters of several adjectival descriptors for each factor. Table 3 depicts words representative of factor clusters. The Kuder-Richardson 8-method was used to determine the reliability of the 16-PF. Table 4 reflectsthereliabilityfigure for eachfactor.

3.2.5 Procedure

The Discus and the 16-PF were administered by trained and registered psychologists. The marking and interpretation of the 16-PF questionnaires were done manually by the researcher, a registered psychologist, and by psychologists from the HSRC.

3.2.6 Statistical Analysis

Pearson's product-moment correlation coefficient was computed, using the SAS-system to determine a correlation between the identified fifteen 16-PF factors and the Discus dimensions.

3.2.7 Results

The 120 questionnaires that were returned were statistically analysed. Scores for each of the fifteen 16-PF factors as well as the Discus dimensions were correlated. The individual Discus variable scores were then correlated with all the fifteen factors of the 16-PF, resulting in significant correlations at 1% level of significance and at 5% level of significance. The results of these findings are reflected in tables 5A (p-values) and 5B (r-values).

TABLE 5A: COMPARISON BETWEEN THE DISCUS DIMENSIONS AND THE FIFTEEN FACTORS OF THE 16-PF. PEARSON'S PRODUCT-MOMENT CORRELATION COEFFICIENT
(5% LEVEL OF SIGNIFICANCE)
(1% LEVEL OF SIGNIFICANCE)

p-values

Discus Dimension

16-PF Factor

1% Level ofSignificance
(99%)

16-PF Factor

5% Level ofSignificance
(95%)

Dominance

Q1
X (Q2, E)
X (G, L, Q1)

0.0004
0.0003
0.0083

E
Q2
-I

0.0141
0.0331
0.0110

Influence

A
-Q2
H

0.0011
0.0011
0.0070

F
-Q3

0.0146
0.0426

Steadiness

None

None

-E
-Q1

0.0363
0.0231

Compliance

-E
Q2
-F
-H

0.0095
0.0092
0.0045
0.0025

-G
O

0.0244
0.0446

TABLE 5B: COMPARISON BETWEEN THE DISCUS DIMENSIONS AND THE 16-PF FACTORS

r-values

Discus Dimension

16-PF Factor

1% Level of Significance

r-values

16-PF Factor

5% Level of Significance

r-values

Dominance

Q1
X (Q2, E)
X (G, L, Q1)

0.32128
0.32846
0.24180

E
Q2
-I

0.22442
0.19551
0.23320

Influence

A
-Q2
H

0.29825
0.29593
0.24791

F
-Q3

0.22519
0.18776

Steadiness

None

None

-E
-Q1

0.19215
0.20907

Compliance

-E
Q2
-F
-H

0.23678
0.23768
0.26000
0.27583

-G
O

0.20715
0.18529

It was found that the Discus instrument is reliable at a significance level = 0.01.

In the validity exercise it was found that the majority of the factors of the 16-PF show a significant correlation with all four dimensions of the Discus using the p-values. The second order factors of the 16-PF were not used for the purpose of this study.

Suelz (1997) is currently busy with a validation study on the Discus using respondents from the Edgars group in Gauteng as his sample size. He has not yet published his masters thesis. It would be interesting to compare the findings of his research with the findings in this study.

he following recommendations have been identified which could open avenues for further research:

The Zulu questionnaire of the Discus instrument was withdrawn from the study because respondents indicated difficulty in understanding the standard of language used in the questionnaire. It is therefore recommended that the Zulu questionnaire be drafted on the basis of day-to-day language use.

For future research it would be interesting to include other instruments to determine whether Discus also correlates with other instruments. It needs to be pointed out that the study of criterion-related validity is based on the measuring of criteria against each other. If any two criteria are not perfectly matched, the validity analysis may be affected. For this reason, this type of study has always proved difficult.

A factor analysis on the Discus could be undertaken for further research.

It is recommended that the instructions to the questionnaire be amended, so as to reflect a specific role in terms of which the individual must complete the questionnaire. It was found that the general manner in which the instructions are presently set out caused individuals to be uncertain as to the role that was being measured, especially those who had more than one role to fulfil in the work situation.