Use of Artificial Intelligence (AI) for Assessments

The use of Artificial Intelligence (AI) for assessments is a very popular topic today. Hopefully, the discussion will not only help people to better understand AI and its opportunities and potential risks but also brings more awareness to standards for assessment use in general.

Assessments in the Human Resources arena for organizational use have different objectives, but there are always guidelines to follow. The guidelines that most people know about are the Uniform Guidelines on Employee Selection Procedures. Any assessment used for decision-making purposes (selection or hiring, promotion, or other types of decisions such as succession-related) should stand up to the scrutiny of these guidelines. If not, the organization is at risk of legal action and the sometimes very substantial costs that come with it.

Guidelines for Assessments Used for Decision-Making

There is also another set of guidelines to follow for the development and use of selection assessments: Principles for the Validation and Use of Personnel Selection Procedures, published by the Society for Industrial and Organizational Psychology (SIOP). Industrial and Organizational Psychologists are rigorously trained in the development and evaluation of tests, assessments, and other selection procedures that are used to make hiring and promotion decisions. Given their training, I-O Psychologists have been advocating for accuracy and fairness in hiring procedures for decades.

The following table provides a high-level comparison of the Uniform Guidelines and SIOP’s Principles:

Guidelines for the Use of AI-Based Assessments for Employee Selection

In 2023, SIOP published Considerations and Recommendations for the Validation and Use of AI-Based Assessments for Employee Selection. They concluded that AI-based assessments should still be required to meet traditional standards for hiring and assessment procedures, even if the way that those standards are evaluated and met varies slightly. They noted five key criteria for AI-based assessments:

1. AI-based assessments should produce scores that predict future job performance (or other relevant outcomes) accurately. The construct measures should be job-related and be supported by scientific evidence.
2. AI-based assessments should produce consistent scores that measure job-related characteristics (e.g., upon re-assessment).
3. AI-based assessments should produce scores that are considered fair and unbiased. Organizations should conduct bias analysis and ensure equitable treatment.
4. AI-based assessments have unique operational considerations and appropriate use guidelines to recognize. The purpose and functionality of the AI system should be understood by stakeholders.
5. All steps and decisions relating to the development and scoring of AI-based assessments should be documented for verification and auditing. This includes data sources, algorithms, and scoring methods.

These five key criteria are intended to represent the minimal requirements necessary to justify the use of AI-based assessments for hiring and promotion decisions. Anyone using AI in assessments should review SIOP’s 2023 guidelines document and implement the recommendations.

Assessments for Development

Beyond assessments for decision-making purpose of course are assessments for strictly development purposes. This includes assessments such as traditional 360-degree assessments and many other assessments used for purposes such as team building or enhancing individual awareness.

While assessments used strictly for development are not typically the subject of the same scrutiny as assessments used for decision-making, it is best practice to ensure that if you are using any assessment, it has documented reliability, validity, and fairness. Consider the following in support of this statement:

• Why use an assessment at all if it does not reliably measure what it purports to measure?
• Anyone in the field of assessment realizes that there will be some organizations who will use assessments that are “strictly developmental,” regardless of best advice, to make decisions. When organizations do this, they are again taking a legal risk.
• Today there are more variations on traditional assessments that may leave an organization unsure of the guidelines that apply to the use of the assessment. For example, traditional 360-degree assessments are meant to be strictly developmental. Today, however, there are “360s” on the market for performance review purposes. There are also “180s” on the market that organizations use in the performance reviews. Again, these assessments used for any type of decision-making are subject to the guidelines outlined above.
• Finally, while 360-degree feedback assessments were historically meant to be used strictly for individual development, every 360 on the market worth considering for use has documented the reliability, validity, and fairness (lack of bias) of the 360 assessment. Why use it if the feedback is not reliable, valid or related to success in the role, and fair (e.g. what if women consistently get lower ratings from their managers than men?)?

360-Degree and 180-Degree Assessments and the Use of AI

360-degree feedback, also called multi-rater feedback, is a method for collecting feedback usually from managers, peers, direct reports, and others about competencies/capabilities and behaviors important to a person’s role. The person also completes the assessment for themselves. 180-degree feedback is similar but typically includes only a self-assessment and an assessment for a person’s managers. The table below summarizes typical distinctions:

AI in 180-Degree Assessments

AI uses in 180-degree assessments often include:

1. Performance trend analysis – AI can track performance over time using historical data and highlight improvements or declines
2. Goal Alignment – AI can compare individual goals to organizational goals and suggest adjustments
3. Feedback Summarization – AI can summarize manager comments into actual insights for the employee
4. Predictive Insights – AI can forecast future performance or promotion readiness based on current data (if an organization is using AI this way, it must be able to support and defend this use – see SIOP guidelines on use of AI-based assessments)

AI in 360-Degree Assessments

360-degree assessments involve multiple feedback sources, making them more complex than 180-degree assessments. AI can help in various ways including:

1. Automating feedback collection – AI can streamline surveys and reminders across various stakeholders; Natural Language Processing (NLP) can analyze open-ended responses for sentiment and themes
2. Analyzing multi-source data – AI can detect patterns across feedback from managers, peers, direct reports, and others; it can identify discrepancies (e.g., self-versus-manager ratings) and flag areas needing attention
3. Personalized development insights – AI can generate tailored development plans based on feedback trends; it can suggest learning resources or coaching based on competency gaps
4. Bias deduction – AI can help detect and reduce bias in feedback by analyzing language and rating patterns

Ethical Considerations in the Use of AI

When incorporating the use of AI into any assessment, there are several critical ethical considerations. Some of these points overlap with the guidelines provided by SIOP for the use of AI in assessments. These include:

1. Data Privacy and Security
   • Collect only Necessary Data: avoid gathering excessive personal information
   • Secure Storage: Use encryption and access controls to protect feedback data
   • Anonymity: Ensure feedback providers remain anonymous when appropriate to encourage honesty and protect identities
2. Fairness and Bias Mitigation
   • Audit AI models: Regularly test for bias based on gender, race, age, or other protected characteristics
   • Diverse training data: Use inclusive datasets to train AI systems
   • Human oversight: Include human reviewers in decision-making processes to catch unintended bias
3. Transparency and Explainability
   • Clear communication: Inform participants how AI is used in the assessment process
   • Explainable outputs: Ensure AI-generated insights can be understood and justified
   • Right to challenge: Allow individuals to question or appeal AI-generated feedback or decisions
4. Consent and Autonomy
   • Informed consent: Get explicit permission from participants before using AI tools
   • Opt-out options: Allow individuals to opt out of AI-driven assessments if they prefer traditional methods
5. Purpose Alignment
   • Use AI to support, not replace: AI should enhance human judgment, not override it
   • Focus on development: Ensure assessments are used for growth and improvement, not punitive measures
6. Continuous Monitoring and Improvement
   • Feedback loops: Regularly collect feedback on the AI system’s performance and impact
   • Update models: Refine AI tools based on new data, ethical standards, and user input

Summary

This article is not meant to provide an exhaustive overview of the use of AI in assessments. Rather, it is meant to cover the more important points to allow organizations and users to critically develop, review, and implement AI-based assessments. Future papers will expand on this initial overview.

If you want to learn more about how Assessments International is integrating AI into The PROFILOR® contact us today to schedule a meeting: https://www.assessmentsinternationalinc.com/contact-form.