SPECIAL INVESTIGATION

CALIFORNIA DEPARTMENT OF INDUSTRIAL RELATIONS

Compliance Review Unit State Personnel Board June 26, 2018

TABLE OF CONTENTS
Introduction	1
Executive Summary	2
Background	2
Scope and Methodology	3
Findings and Recommendations	6
Departmental Response	10
SPB Reply	10

INTRODUCTION

Established by the California Constitution, the State Personnel Board (the SPB or Board) is charged with enforcing and administering the civil service statutes, prescribing probationary periods and classifications, adopting regulations, and reviewing disciplinary actions and merit-related appeals. The SPB oversees the merit-based recruitment and selection process for the hiring of over 200,000 state employees. These employees provide critical services to the people of California, including but not limited to, protecting life and property, managing emergency operations, providing education, promoting the public health, and preserving the environment. The SPB provides direction to departments through the Board’s decisions, rules, policies, and consultation.

Pursuant to Government Code section 18661, the SPB’s Compliance Review Unit (CRU) conducts compliance reviews of appointing authority’s personnel practices in five areas: examinations, appointments, equal employment opportunity (EEO), personal services contracts (PSC’s), and mandated training, to ensure compliance with civil service laws and board regulations. The purpose of these reviews is to ensure state agencies are in compliance with merit related laws, rules, and policies and to identify and share best practices identified during the reviews.

Effective July 1, 2012, the Governor's Reorganization Plan Number One (GRP1) of 2011 consolidated all of the functions of the Department of Personnel Administration and the merit-related operational functions of the State Personnel Board (SPB) into the California Department of Human Resources (CalHR). Pursuant to Government Code Section 18502(c), CalHR and SPB may “delegate, share, or transfer between them responsibilities for programs within their respective jurisdictions pursuant to an agreement.” CalHR and SPB, by mutual agreement, expanded the scope of items reviewed by the CRU beyond merit-related issues into more operational practices that are delegated to departments, and for which CalHR provides policy direction. Many of these delegated practices are cost drivers to the state and were not being monitored on a statewide basis.

As such, SPB also conducts compliance reviews of appointing authorities’ personnel practices to ensure that state departments are appropriately managing the following non- merit-related personnel functions: compensation and pay, leave, and policy and processes. These reviews will help to avoid and prevent potential costly litigation related to improper personnel practices, and deter waste, fraud and abuse.

The SPB conducts these reviews on a three-year cycle.

The CRU may also conduct special investigations in response to a specific request or when the SPB obtains information suggesting a potential merit-related violation.

EXECUTIVE SUMMARY

In 2015, the State Personnel Board’s (SPB or the Board) Appeals Division (AD) received a merit complaint from a candidate appealing his unsuccessful score in the Associate Safety Engineer (ASE) (Elevators) examination administered by the Department of Industrial Relations (DIR).

Upon review, the AD determined there were significant irregularities in the ASE (Elevators) examination, and directed the CRU to complete a special investigation into DIR’s examination processes. Additionally, the DIR was ordered to utilize the CalHR for the construction of all DIR examinations for a period of two years.

Since March 9, 2017, the CalHR has provided ongoing review, guidance, support, and training regarding DIR’s examination development and validation processes to ensure adherence to the merit rules, regulations, and statutes.

On September 11, 2017, the SPB began a special investigation of DIR’s examination processes. The results from this investigation were compiled and the findings have been grouped into two main categories: 1) Examination Administration, which refers to how the examination itself was administered by the panel members, and 2) Examination Process, which refers to the manner in which staff processed and maintained documents and other information relative to the examination.

CRU found that DIR improperly administered an open/non-promotional Associate Safety Engineer (ASE) examination due to inconsistent probing by panel members. CRU also found that DIR failed to keep sufficient and accurate documentation throughout the examination process.

BACKGROUND

The Department of Industrial Relations (DIR) was established in 1927. Its mission is to improve working conditions for California’s wage earners and to advance opportunities for profitable employment in California. The department is comprised of the Division of Labor Standards Enforcement, Division of Occupational Safety and Health, Division of Workers’ Compensation, Division of Apprenticeship Standards, Boards, Commissions and Taskforce. DIR administers and enforces laws governing wages, hours and breaks,

overtime, retaliation, workplace safety and health, apprenticeship training programs, and medical care and other benefits for injured workers. As of April 2018, the DIR employs over 2,800 employees to carry out the mission of the department, divisions, boards and programs.

SCOPE AND METHODOLOGY

The CRU reviewed the following 32 examinations administered by DIR between July 1, 2016 and March 31, 2017: 13 Training & Experience (T&E) examinations, 12 Qualifications Appraisal Panel (QAP), 4 Career Executive Assignments (CEA) and 3 Statement of Qualifications (SOQ). The 32 examinations were for 22 different classifications. Twenty-one classifications were civil service classifications; therefore, the examinations for those classifications required a valid job analysis.

The CRU examined examination plans and bulletins, 511B’s, job analysis reports, applications, orientation materials, chairperson’s scripts, scoring sheets, raters’ notes for competitors, supplemental applications, examination items, scoring reports, audio tapes, examination validation documentation, exam development and administration procedures, and candidate lists.

During the period under review, DIR conducted 44 examinations. The CRU reviewed 32 of those examinations, which are listed below:

Classification	Exam Type	Exam Components	Final File Date	No. of Apps	

Apprenticeship Consultant	Open/Non Promotional	Training and Experience (T&E)1	
08/09/16	
20	

Associate Personnel Analyst	Departmental Promotional	Qualification Appraisal Panel (QAP)2	
12/23/16	
7	

Associate Safety Engineer	Open/Non Promotional	
QAP	
12/27/16	
131	

1 The training and experience (T&E) examination is administered either online or in writing, and asks the applicant to answer multiple-choice questions about his or her level of training and/or experience performing certain tasks typically performed by those in this classification. Responses yield point values.
2 The qualification appraisal panel (QAP) interview is the oral component of an examination whereby competitors appear before a panel of two or more evaluators and respond to verbally to oral interview questions. Candidates are rated and ranked against one another based on their responses to questions that assess their ability to perform in the job classification. The same questions are asked of every candidate and the same rating criteria are applied.

Classification	Exam Type	Exam Components	Final File Date	No. of Apps	

Associate Safety Engineer	
Promotional	
QAP	
04/15/16	
30	
Associate Safety Engineer (Mining and Tunneling)	
Open	
QAP	
07/28/16	
10	
Associate Safety Engineer (Pressure Vessels)	Open/Non Promotional	
QAP	
11/18/16	
17	
Associate Safety Engineer (Amusement Rides)	Open/Non Promotional	
QAP	
11/18/16	
14	

CEA A, Chief of Human Resources	
Open	
CEA	
12/08/16	
1	
CEA A, Chief of Medical Services Administration	
Open	
CEA	
10/31/2016	
1	

CEA B, Chief of Public Works	
Open	
CEA	
12/22/16	
4	

CEA B, Chief of Self Insurance Plan	
Open	
CEA	
10/11/16	
2	

Deputy Labor Commissioner I	Open/Non Promotional	
T&E	
08/15/16	
79	

Deputy Labor Commissioner II	Open/Non Promotional	
T&E	
06/30/16	
70	

Deputy Labor Commissioner II	Open/Non Promotional	
T&E	
10/31/16	
33	

Deputy Labor Commissioner III	Open/Non Promotional	
T&E	
07/15/16	
33	

Deputy Labor Commissioner III	Open/Non Promotional	
T&E	
11/15/16	
47	

Industrial Relations Representative	Open/Non Promotional	
T&E	
07/29/16	
139	

Industrial Relations Representative	Open/Non Promotional	
T&E	
11/30/16	
105	

Classification	Exam Type	Exam Components	Final File Date	No. of Apps	

Junior Safety Engineer	Open/Non Promotional	
T&E	
11/04/16	
248	

Presiding Workers' Compensation Judge	Open/Non Promotional	Statement of Qualifications (SOQ)3	
07/01/16	
32	

Presiding Workers' Compensation Judge	Department Promotional	
T&E	
04/29/16	
7	

Principal Safety Engineer (Industrial)	Departmental Promotional	
QAP	
08/05/16	
19	

Research Analyst I (Economics)	Open/Non Promotional	
QAP	
11/18/16	
9	

Senior Apprenticeship Consultant	Open/Non Promotional	
SOQ	
06/03/16	
16	
Senior Safety Engineer (Amusement Rides)	Department Promotional	
QAP	
08/05/16	
14	

Senior Safety Engineer (Elevators)	
Promotional	
QAP	
08/05/16	
9	

Senior Safety Engineer (Industrial)	Departmental Promotional	
QAP	
12/06/16	
73	
Senior Safety Engineer (Pressure Vessel)	Departmental Promotional	
QAP	
12/16/16	
8	

Workers' Compensation Consultant	Open/Non Promotional	
T&E	
06/30/16	
26	

Workers' Compensation Consultant	Open/Non Promotional	
T&E	
09/16/16	
12	

Workers' Compensation Consultant	Open/Non Promotional	
T&E	
11/18/16	
12	

Workers' Compensation Judge	Open/Non Promotional	
SOQ	
04/08/16	
21	

3 In a statement of qualifications (SOQ’s) examination, applicants submit a written summary of their qualifications and experience related to a published list of desired qualifications. Raters, typically subject matter experts, evaluate the responses according to a predetermined rating scale designed to assess their ability to perform in a job classification, assign scores and rank the competitors in a list.

FINDINGS AND RECOMMENDATIONS

The CRU found the following deficiencies in the DIR’s examination administration process:

FINDING NO. 1 – Improper Probing by Panel Members Had a Material Effect on the Candidates’ Scores and Ranks

DIR administered an open, non-promotional examination for Associate Safety Engineer with a final filing date of December 27, 2016. The examination method used was a Qualifications Appraisal Panel (QAP). The candidates were scored based on their responses to six structured questions, some of which had multiple parts. Each structured question had a set of suggested responses and rating criteria. Each suggested response was assigned one point, so that if the candidate mentioned the information in the suggested response, a point would be awarded. The number of points available for each set of suggested responses varied from seven to 28 points depending on the question. The sum of the points for each question determined the score for that question. The sum of the scores for each question determined the candidate’s score on the examination. The scores for all the candidates were converted to list scores, which were banded into ranks one through eight as follows:

Number of Candidates	
List Score	
Rank	

1	
91	
1	

1	
88	
2	

1	
85	
3	

7	
82	
4	

9	
79	
5	

12	
76	
6	

11	
73	
7	

9	
70	
8	

14		
Disqualified	

A total of 65 applicants took the examination, with 51 candidates receiving a passing score of 70 or higher. The CRU listened to all 65 audio tapes from the oral exams, each 45 minutes in length, and found improper probing in the panel members’ questioning for some candidates that resulted in additional points being awarded to those candidates’ scores. Specifically, the CRU determined that eight candidates were awarded more points after the question(s) were clarified or leading question(s) were asked. The increase in the eight scores impacted 46 or 90% of the candidates’ rankings on the certification list. CRU determined that, without the improper probing, the top score, rank 1, would have been 88 rather than 91. As a result, 42 candidates were pushed down a rank, one candidate was elevated two ranks, three candidates were elevated one rank, and five candidates remained in their original ranks.

Examinations for the establishment of eligible lists shall be competitive and of such character as fairly to test and determine the qualifications, fitness, and ability of competitors actually to perform the duties of the class of position for which they seek appointment. (Gov. Code, §18930). Additionally, “all QAP interviews for an examination shall be structured and use the same pre-determined, job related questions. The QAP shall not rephrase or clarify any candidate inquiries.” (Cal. Code of Regs., tit.2, §195.1, subd. (b).) In order to administer a competitive and fair exam, all candidates must be provided the same opportunity for success. In QAP exams, this means that all candidates must be given the same pre-scripted exam questions, and their responses must be measured against the same prepared suggested responses and scoring criteria. By assisting some candidates to give a more comprehensive response, panel members are providing those candidates with an unfair advantage over candidates who were not provided the assistance. Each QAP panel must have a certified Chairperson who has completed training on how to properly conduct a QAP exam. The Chairperson is responsible for ensuring that the testing experience is the same for all the candidates. (Cal. Code of Regs., tit. 2, §195, subd. (c).) One of the main functions of the Chairperson is to ensure that no improper probing occurs during the exam.

Because it has been determined that the improper probing resulted in an increase in some candidates’ scores and a correlated decrease in other candidates’ rankings, CRU

found that the improper practice had a material effect on the eligibility list. Because of the gravity of DIR’s errors, CRU recommends invalidating the exam within 60 days.

SPB has the discretion to void any appointments made from the eligibility list of an invalidated exam. In this case, however, after careful review, it has been determined that it would not be in the best interests of the State to void the 22 appointments made from the list. Significantly, the improprieties identified in this investigation were committed by the department and not any of the candidates. The department has invested extensive training in each of the incumbents, which would be lost if the employees were to lose their jobs. Associate Safety Engineers with DIR are responsible for conducting safety inspections of California’s worksites. They ensure that California’s factories, construction sites, warehouses, and other worksites are free of physical and chemical hazards. The work performed by incumbents in the classification is vital to the safety of employees throughout various industries in California. DIR has experienced significant challenges in recruiting and retaining a consistent workforce in this division, and voiding these 22 appointments would only compound those challenges, not to mention the hardship it would cause to the employees who did nothing wrong. Additionally, while the improprieties had a material effect on the results of the exam, candidates in all ranks eventually became reachable, and therefore, eligible for appointment. In fact, two candidates in rank seven have been appointed.

Furthermore, there was no evidence of pre-selection. There was no indication that the probing was geared toward known candidates in order to enhance their scores to ensure their eligibility for appointment. The probing occurred with both internal and external candidates. In listening to the recordings, CRU determined that the additional questioning appeared to be done to legitimately explore the extent of the candidate’s knowledge, skills, and abilities, and not to suggest specific responses.

FINDING NO. 2 – DIR Failed to Keep Sufficient and Accurate Documentation Throughout the Examination Process

The CRU found several violations of recordkeeping rules and procedures. The violations fall into three categories:

No.	Issue	Impact	
1	DIR did not properly maintain all applications submitted for examinations for the required period of time.	The CRU is unable to determine if all applicants met the MQs for admission to the examination.	

2	DIR did not have documentation linking the examination items to the job analyses and class specifications.	After reviewing the job analyses and exam items, the CRU determined that the examination items link to the job analyses and class specifications.
However, the DIR failed to document these linkages during the examination development process.	
3	One candidate’s QAP audio tape was missing and DIR was unable to provide for CRU to review.	This is a technical violation because the audio tapes are backup documentation for the primary rating sheets and raters’ notes.	

The SPB’s regulations provide that appointing powers shall retain the following records for a minimum of five years from the date of creation of the record: examination records including, but not limited to, the examination bulletin, examination planning documents and/or forms, job analysis, list of accepted and rejected candidates, all applications received with accepted and rejected notations, minimum qualification verifications, any notices sent to candidates, examination questions, model answer sheets, candidate answer sheets, rating criteria, rating sheets and rater’s notes for each candidate. (Cal. Code Reg., tit. 2, §26 subd. (a) (2).)

Without adequate documentation, the CRU is unable to determine if any merit issues occurred during the development, administration and scoring phases of the examination. Best practices require that the entire examination process be sufficiently documented from the beginning to the end. Failure to do so places the examination in jeopardy of being invalidated and the department in jeopardy of losing its delegated authority to develop and administer examinations. Furthermore, in the event of an appeal of the examination, the department cannot adequately defend itself before the Board and/or in a court case.

Because this examination was conducted prior to the Board revoking DIR’s delegated authority to conduct its own exams, and DIR is currently conducting its exams under the supervision of CalHR, no further recommendation is being made with respect to DIR’s delegation.

However, it is recommended that, as part of CalHR’s supervision of DIR’s examination function, DIR staff involved in administering QAP exams undergo additional training on the proper procedures. Specifically, it is recommended that DIR’s Chairpersons be re- certified  to  gain  a  better  understanding  of  their  role  and  responsibilities.  It  is also

recommended that DIR exam staff be trained on the documentation and recordkeeping procedures related to examinations.

Additionally, it is recommended that DIR change its examination instrument. The QAP exam is a costly, time-consuming, and inefficient method of examining. ASE is a journey- level classification that requires a background in workplace safety, which can be demonstrated through an online training and experience (T & E) exam. In T & E exams, candidates are rated based on their self-reported work experience, education and training. T & E exams are accessible to the candidates 24/7 and provide the department with a fresh and more robust candidate pool on an ongoing basis. After the examination phase, eligible candidates are scheduled for job interviews, during which the candidate’s job-specific knowledge, skills, and abilities can be more fully explored without restrictions on probing and leading questions.

Within 60 days of the Executive Officer’s approval of these findings and recommendations, the DIR must present to the Board its corrective action plan that addresses the actions the department will implement to ensure that the merit process is followed in future exams.

DEPARTMENTAL RESPONSE

The DIR’s response is attached. In summary, the DIR is committed to following all of the SPB’s recommendations related to invalidating the ASE examination, providing staff training on proper exam procedures, documentation and record keeping, and changing its examination instrument.

SPB REPLY

The SPB appreciates the DIR’s commitment to improve its examination processes and looks forward to DIR’s corrective action plan.

STATE OF CALIFORNIA	EDMUND G. BROWN JR., Governor

DEPARTMENT OF INDUSTRIAL RELATIONS
Andre Schoorl, Acting Director
Office of the Director
1515 Clay Sh·eet, 17th Floor
Oakland, CA 94612
Tel: (510) 286-7087 Fax: (510) 622-3265

July 16, 2018

Suzanne M . Ambrose Executive Officer State Personnel Board
801 Capitol Mall, MS #22
Sacramento,  CA 95814

RE: Special Investigation of the Department of Industrial Relation's (DIR) Examination Processes

Dear Ms. Ambrose,

I would like to thank the Compliance Review Unit for their time, insight, and feedback during the review process. I take these findings very seriously and I am committed to revising DIR's processes and procedures to ensure future DIR exams are administered in accordance with all civil service laws, rnles, and regulations.

I agree with the two findings for the Associate Safety Engineer exam given in 2017: (1) improper probing by panel members had a material effect on the candidates' scores and ranks; and (2) failure to keep sufficient and accurate documentation through-out the examination process and offer the following responses:

Recommendation 1. The Compliance Review Unit recommends invalidating the Associate Safety Engineer exam within 60 days.

The eligibility list for the 2017 Associate Safety Engineer exam was abolished; invalidating the exam.

Recommendation 2: DIR staff involved in administering Qualification Appraisal Panel exams undergo additional training on the proper procedures. Specifically, it is recommended that DIR' s Chairpersons be re-certified to gain a better understanding of their role and responsibilities. It is also recommended that DIR exam staff be trained on the documentation and recordkeeping procedures related to examinations.

Exam Unit staff have been re-enrolled in California Department of Human Resources (CalHR) training courses that are taking place in July and August 2018. The courses include proper exam procedures and the roles and responsibilities of chairing Qualification Appraisal Panel exams.

I Page2

DIR is cmTently reviewing records of all exams given year-to-date in 2018. We are working with CalHR to schedule additional re-training on the documentation and recordkeeping procedures related to examinations.

Recommendation 3: DIR change its examination instrument.

The Qualification Appraisal Panel approach for testing is being eliminated as a standard departmental examination practice. Cun-ent exams that have been planned as Qualification Appraisal Panel exams will be re-developed under the supervision of CalHR into training and experience(T&E) exams.

Thank you and your staff for the extensive work and recommendations to improve the department's administration of exams to ensure a fair and competitive process for all candidates. If you have any questions or concerns, please contact me at the number listed above.

Sincerely,
ANDRESCHOORL
Acting Director