481 McLaws Circle, Suite 1
Williamsburg, Virginia 23185
Diagram 2: How much improvement do clients experience:
RANDY WALTON, PH.D. - EFFECTIVENESS AND OUTCOME DATA
I have been obtaining data regarding my treatment outcomes since 2008. I use this data to improve treatment outcomes with each client, and to evaluate my overall effectiveness and outcomes as a therapist.
How do I obtain the data?
The data is obtained through the use of brief (1-2 minutes) rating scales that are completed each session by the client. One rating scale is completed at the start of the session to capture changes/improvements between sessions, while a second one is completed at the end of the session to provide feedback and offer suggestions about what was helpful and not helpful. The client and therapist review and discuss the rating scale scores during each session.
Why do I obtain and report this data?
The review and discussion of the rating scale scores helps guide treatment. It allows us to identify and reinforce positive changes, and make adjustments to treatment if improvements are not reported.
These same rating scale scores also provide the data reported below. This data is used to evaluate the effectiveness and outcomes of the counseling services I provide. Reporting this data allows clients, potential clients, and stakeholders to compare and make more informed choices regarding their therapist or treatment provider.
The scores are only reported in aggregate form, and no individual clients are ever identified.
Two types of effectiveness data are reported for my clients: how many people are helped (Diagram 1), and how much they are helped (Diagram 2).
Diagram 1: How many people are helped:
68% of Dr. Walton's counseling clients achieve their “target” score compared to 50% of counseling clients in a large national sample.
See below for more detail and statistical information regarding Diagram 1
Diagram 2. How much improvement is experienced by Dr. Walton's clients
Diagram 2 presents a comparison of the "effect sizes" between clients in a large normative sample (ASIST for the Professional, v. 2.14) and clients in treatment with Dr. Walton. Effect size provides information about the size, magnitude, or meaningfulness of the change people experience from counseling. In other words, how much the counseling has helped, rather than simply indicating whether it helped or not. The effect size in Diagram 2 is represented by the percentage of improvement or relief reported by clients.
A simple example can help illustrate: If two pain medications have been shown to relieve pain, but medication A provides much more pain relief than medication B, then medication A is more effective at relieving pain than medication B. In this case the effect size of medication A will be larger than for medication B.
In order to provide a fair comparison, the reported effect size for Dr. Walton's clients is compared to the effect size for clients who receive counseling with other therapists (i.e., clients in the normative database for ASIST for the Professional, v 2.14). This is more meaningful in determining effectiveness than a comparison of Dr. Walton's clients with individuals who are not in counseling at all, since research demonstrates that most counseling is likely to provide relief from distress (similar to the way both medications in the example provide more relief than no treatment at all). Comparing the effect sizes between different therapists can help determine which ones are relatively more effective.
How effect size is calculated:
The effect size is calculated by first subtracting each client's predicted outcome or “target” score from their actual current score. This difference between each client's actual score and target score is known statistically as a “residualized change score”. The residualized change score is then divided by the standard deviation of the rating scale which the client completed. (The standard deviation provides information about the probability that a single measurement will vary from the mean.) Dividing the residual change score by the standard deviation helps account for normal variations in scores that might occur by chance, and allows comparison between different clients and different measures. The end result of these statistical calculations is called the effect size. A positive effect size means that a client is experiencing more improvement than expected (based upon the target score), and the larger the effect size, the greater the amount of improvement.
The effect size in Diagram 2 is reported as a percentage, since this is easily understood by most people. Alternatively, the effect size can be reported as the number of standard deviations above or below the mean of the normative sample (ASIST for the Professional, v 2.14). If effect size is reported in terms of standard deviations, the amount of improvement reported by Dr. Walton's clients is +0.31, or 0.31 standard deviations above the mean for the normative sample. The normative sample is represented by a normal distribution, and the mean amount of improvement for the normative sample is 50%. In a normal distribution, 0.31 standard deviations to either side of the mean is equivalent to 12% above or below the mean. By adding 12% (since effect size for Dr. Walton's clients is a positive effect size) to 50% (the mean), the percentage of improvement reported by Dr. Walton's clients is 62%.
Translating this technical information into less technical terms, the average client in the large counseling sample experiences 50% improvement or relief, and the average client of Dr. Walton experiences about 62% improvement or relief.
Based upon effectiveness data collected since 2008,
Dr. Walton's clients benefit more than the average counseling client.
The data presented above was collected through March 1, 2018.
RANDY WALTON, PH.D. - EFFECTIVENESS AND OUTCOMES: DETAILS AND STATISTICS
Diagram 1. How many people are helped:
Diagram 1 compares the percentage of counseling clients who achieve a "target” score, sometimes called a "benchmark" score or "predicted" score. The data in Diagram 1 is based on the scores reported by counseling clients on a brief rating scale completed at the beginning of each counseling session. The target score is a score that we "aim for", and it represents the expected or predicted improvement for a person based upon comparison with a large sample of counseling clients. The target score varies for each person depending on that person's level of distress when they start treatment. Reasonable or satisfactory improvement for someone who is severely distressed at the start of treatment is different than for someone reporting less severe distress at the start of treatment.
A simple example can illustrate: consider two people who both want to reduce their coffee intake due to sleeplessness and anxiety. If one person drinks 20 cups of coffee per day, and the other person drinks 5 cups of coffee per day, it might not be realistic or useful to set the same "target” score for both people. Reasonable, satisfactory improvement for the 20 cup person might be a reduction of coffee consumption to five cups per day, while reasonable, satisfactory improvement for the 5 cup person might be two cups per day.
Of course in the real world everyone with the same intake score will not achieve exactly the same final score. Some will score higher and some will score lower. However, the target score represents a reasonable estimate of expected improvement, and provides a helpful reference point to determine whether any given person is improving more than usual or less than expected. Adjustments to treatment can be made based on this information.
How target score is calculated:
A person's target score is directly dependent on his or her initial, or intake, score. The person's intake score is matched with a large normative sample of clients (ASIST for the Professional, v. 2.14) who reported that same intake score. The "average" improvement for this large sample of clients is calculated using a linear regression equation, and this "average" improvement or relief becomes the target score for everyone with that initial or intake score.
In more technical terms, the target/benchmark score is based on severity-adjusted change scores which are calculated from the generalized linear model (GLM) of a normative client database (ASIST for the Professional, v. 2.14). The GLM is a statistical regression formula which is used to determine average final scores for each possible intake score. The GLM generalizes linear regression by allowing the linear model to be related to the response variable via a link function, and by allowing the magnitude of the variance of each measurement to be a function of its predicted value. The result is a normal distribution of "final" scores for each intake score, and the target score represents the 50th percentile of this normal distribution. Determining the target score in this way allows statistical prediction of a person's final score based only upon that person's intake score.
While 50% of counseling clients typically achieved or exceeded their target score in a large national sample, 68% of Dr. Walton's clients achieved or exceeded their target score
Dr. Walton's counseling clients report 62% improvement compared to 50% improvement or relief reported by counseling clients in a large national sample
See below for more detail and statistical information regarding Diagram 2