2018-2019 Graduate Students Assessment Results

Department of Animal and Range Sciences

by Dr. Lance McNew and Dr. Sarah McCoski

The Department of Animal and Range Sciences (hereafter “department”) offers a Master of Science degree in Animal & Range Sciences and a Doctor of Philosophy degree in Animal & Range Sciences. Both the M.S. and Ph.D. degrees require that the student choose either an Animal Science emphasis or a Range Science emphasis.  In addition, Ph.D. students may opt to enroll in the interdisciplinary Ecology and Environmental Sciences degree program. Ph.D. students enrolled in this degree program are affiliated with a home department that corresponds to that of their major faculty advisor, and are required to meet standard departmental requirements.

Department faculty met during the Spring of 2018 to formulate a plan to assess the department’s graduate program.  A final plan was approved in May of 2018 and immediately implemented.  As required by the plan, the department is required to report on assessment activities and critically evaluate the department’s graduate program.  The following documents serve as our assessment for the time period January 2018 – August 2019.

The Department identified the following desired learning outcomes for graduate students completing Master of Science and Ph.D. degrees:

For M.S. students:

1. Conduct research resulting in an original thesis or dissertation
2. Demonstrate mastery of subject content knowledge and research/critical inquiry methodology
3. Demonstrate effective written communication of substantive content
4. Demonstrate effective oral communication of substantive content
5. Be able to conduct scholarly or professional activities in an ethical manner

For Ph.D. students:

1. Produce and defend an original significant contribution to knowledge
2. Demonstrate mastery of subject content knowledge and research/critical inquiry methodology
3. Demonstrate excellence in written communication of substantive content
4. Demonstrate excellence in oral communication of substantive content
5. Be able to conduct scholarly and professional activities in an ethical manner
6. Demonstrate professionalization into the field of study as demonstrated through publications, presentations, funded fellowships, professional association activities, professional experience, etc.

To evaluate departmental success in meeting desired learning outcomes, we collected student data from the following sources:

Student Performance: Data Sources

M.S. Programs

PhD. Programs

Results:

During January 2017 ­– August 2019, 17 M.S. and 3 Ph.D. students attempted to defend graduate degrees.  All graduate students were successful. Five M.S. students withdrew from the graduate program during the reporting period. Six Ph.D. students successfully completed comprehensive exams.

M.S. Students

Learning Outcome 1: Conduct research resulting in an original thesis or dissertation.

During January 2017 – August 2019, 17 students attempted to defend an M.S. thesis describing original research; all 17 produced an original thesis and successfully defended.  Graduate students successfully defending theses and earning M.S. degrees during January 2017 – August 2017 are listed in Table 1.

Learning Outcome 2: Demonstrate mastery of subject content knowledge and research/critical inquiry methodology

Beginning in 2018, graduate student advisory committees completed a Thesis and Defense Assessment (Appendix A), where each member of the graduate supervisory committee ranked the student’s performance relative to the learning outcome from 1 (Unacceptable) to 4 (Exceeds standards).  Scores were tabulated and averages calculated for each student.  Assessments were completed for eleven M.S. students defending during January 2018­—August 2019. 

The mean (± SD) score relative to Learning Outcome 2 was 3.4 (0.6).  Ten (10) of 11 students received average scores ≥3, indicating that 91% of the M.S. students defending during January 2018 – August 2019 met departmental standards relative to the learning outcome.  Thus, we exceeded the departmental response threshold that ≥ 80% of M.S. students will be ranked at a 3–4 level in subject content knowledge.

Learning Outcome 3: Demonstrate effective written communication of substantive content

Learning Outcome 3 was assessed using two data sources. First, graduate student supervisory committees completed a Thesis and Defense Assessment (Appendix A), where each member of the graduate supervisory committee ranked the student’s performance relative to the learning outcome from 1 (Unacceptable) to 4 (Exceeds standards).  The second data set in composed of writing assignments provided by graduate students as part of ARNR 507: Research Methods.  Each semester of course offering, two members of the faculty who were not the primary instructor for the course scored a random sample of writing assignments from students taking the class for the second or third time (i.e., near the end of their graduate program). We used the rubric presented in Appendix B to evaluate students relative to the learning outcome.

The mean (± SD) score collected by graduate committees at defenses relative to Learning Outcome 3 was 3.1 (0.5).  Eight (8) of 11 students received average scores of ≥3, indicating that 73% of the M.S. students met departmental standards relative to the learning outcome.  During January–August 2018, we did not meet our response threshold that ≥ 80% of students will be ranked at a 3–4 level in written communication. 

The Graduate Assessment Committee independently evaluated the writing assignments of five M.S. students enrolled in their second semester of ARNR 507.  Students were ranked based on four criteria and average scores (e.g., 1 = unacceptable, 4 = exceeds standards) across criteria calculated for each student.  Average writing scores for M.S. students was 3.1 (0.5).  Four of 5 students evaluated had overall scores ≥3, indicating that 80% of students sampled met departmental standards for effective written communication.

Learning Outcome 4: Demonstrate effective oral communication of substantive content

Learning Outcome 4 was assessed using two data sources. First, graduate student supervisory committees completed a Thesis and Defense Assessment (Appendix A), where each member of the graduate supervisory committee ranked the student’s performance relative to the learning outcome from 1 (Unacceptable) to 4 (Exceeds standards).  The second data set in composed of independent evaluations of oral presentations given by graduate students as part of ARNR 507: Research Methods.  Each semester of course offering, two members of the faculty who were not the primary instructor for the course scored a random sample of writing assignments from students taking the class for the second or third time (i.e., near the end of their graduate program). We used the rubric presented in Appendix C to evaluate students relative to the learning outcome.

The mean (± SD) score collected by graduate advisory committees at M.S. defenses relative to Learning Outcome 4 was 3.5 (0.5).  All 11 M.S. students defending during the reporting period received average scores of ≥3, indicating that 100% of the M.S. students met departmental standards relative to proficiency in oral communication. 

The Graduate Assessment Committee independently evaluated the oral presentations of 7 M.S. students enrolled in ARNR 507 during the January 2018 – May 2019). Average (SD) scores for effectiveness in the oral communication of student research was 3.6 (0.3).  All 7 students (100%) students received overall oral presentation scores ≥3.0 (range = 3.1–3.9).

Overall, both lines of evidence indicate that the department met the desired response threshold of M.S. student proficiency in oral communication.

Learning Outcome 5: Be able to conduct scholarly or professional activities in an ethical manner

All students received training in scientific ethics as part of ARNR 507.  In addition, graduate student supervisory committees completed a Thesis and Defense Assessment (Appendix A), where each member of the graduate supervisory committee ranked the student’s performance relative to the learning outcome from 1 (Unacceptable) to 4 (Exceeds standards).

The mean (± SD) score collected by graduate committees at defenses relative to Learning Outcome 5 was 3.7 (0.5).  All 11 students defending during the reporting period received average scores of ≥ 3, indicating that 100% of the M.S. students defending during 2018—2019 met departmental standards relative to ethical scholarly and professional activities.

Ph.D. Students

Learning Outcome 1: Produce and defend an original significant contribution to knowledge

During 2017–2018, three students attempted to defend a Ph.D. thesis describing original research, all of which (100%) produced an original dissertation and successfully defended. Graduate students successfully defending dissertations and earning doctoral degrees during January 2017 – August 2017 were Omolola Betiku, Neto Garcia, and Samuel Wyffels (Table 1).

Learning Outcome 2: Demonstrate mastery of subject content knowledge and research/critical inquiry methodology

Beginning in 2018, graduate student advisory committees completed a Thesis and Defense Assessment (Appendix A), where each member of the graduate supervisory committee ranked the student’s performance relative to the learning outcome from 1 (Unacceptable) to 4 (Exceeds standards).  Scores were tabulated and averages calculated for each student.  Assessments were completed for two Ph.D. students defending during January 2018­—August 2019. 

The mean (± SD) score relative to Learning Outcome 2 was 3.4 (0.6).  Both students received average scores ≥3, indicating that 100% of the doctoral students defending during January 2018 – August 2019 met departmental standards relative to the learning outcome.  Thus, we met our departmental response threshold that ≥ 80% of students will be ranked at a 3–4 level in subject content knowledge. 

Learning Outcome 3: Demonstrate effective written communication of substantive content

Learning Outcome 3 was assessed using two data sources. First, graduate student supervisory committees completed a Thesis and Defense Assessment (Appendix A), where each member of the graduate supervisory committee ranked the student’s performance relative to the learning outcome from 1 (Unacceptable) to 4 (Exceeds standards).  The second data set in composed of writing assignments provided by graduate students as part of ARNR 507: Research Methods.  Each semester of course offering, two members of the faculty who were not the primary instructor for the course scored a random sample of writing assignments from students taking the class for the second or third time (i.e., near the end of their graduate program). We used the rubric presented in Appendix B to evaluate students relative to the learning outcome.

The mean (± SD) score collected by graduate committees at defenses relative to Learning Outcome 3 was 3.3 (0.3).  Both Ph.D. students (100%) defending during 2018-2019 received average scores of ≥3.

The Graduate Assessment Committee independently evaluated the writing assignments of three Ph.D. students enrolled in their second semester of ARNR 507.  Students were ranked based on four criteria and average scores (e.g., 1 = unacceptable, 4 = exceeds standards) across criteria calculated for each student.  Average writing scores for Ph.D. students was 3.8 (0.3).  All three students evaluated had scores ≥3, indicating that 100% of students met departmental standards for effective written communication.

Learning Outcome 4: Demonstrate effective oral communication of substantive content

Learning Outcome 4 was assessed using two data sources. First, graduate student supervisory committees completed a Thesis and Defense Assessment (Appendix A), where each member of the graduate supervisory committee ranked the student’s performance relative to the learning outcome from 1 (Unacceptable) to 4 (Exceeds standards).  The second data set in composed of independent evaluations of oral presentations given by graduate students as part of ARNR 507: Research Methods.  Each semester of course offering, two members of the faculty who were not the primary instructor for the course scored a random sample of writing assignments from students taking the class for the second or third time (i.e., near the end of their graduate program). We used the rubric presented in Appendix C to evaluate students relative to the learning outcome.

The mean (± SD) score collected by graduate committees at defenses relative to Learning Outcome 4 was 3.4 (0.4).  Both students defending during the reporting period received average scores of ≥3, indicating that 100% of the Ph.D. students met departmental standards relative to proficiency in oral communication based on graduate committee assessment. 

During three semesters of ARNR 507 (spring 2018 & 2019, fall 2019), we evaluated the oral presentations of two Ph.D. students.  Average scores for effectiveness in the oral communication of student research was 3.4 (0.4).  Both (100%) students received overall average scores ≥3.0.

Overall, both lines of evidence indicate that the department met the desired response threshold of Ph.D. student proficiency in oral communication.

Learning Outcome 5: Be able to conduct scholarly or professional activities in an ethical manner

All students received training in scientific ethics as part of ARNR 507.  In addition, graduate student supervisory committees completed a Thesis and Defense Assessment (Appendix A), where each member of the graduate supervisory committee ranked the student’s performance relative to the learning outcome from 1 (Unacceptable) to 4 (Exceeds standards).

The mean (± SD) score collected by graduate committees at defenses relative to Learning Outcome 5 was 3.7 (0.1).  Both students defending during the reporting period received average scores of ≥ 3, indicating that 100% of the Ph.D. students defending during 2018—2019 met departmental standards relative to ethical scholarly and professional activities.

Learning Outcome 6: Demonstrate professionalization into the field of study as demonstrated through publications, presentations, funded fellowships, professional association activities, professional experience, etc.

The curriculum vitae of two Ph.D. students were collected after their doctoral defense and evaluated for evidence of professionalization into their field of study. 

Publications: On average, Ph.D. students published 1 publication in peer-reviewed and 5 other scientific/professional journals associated with their field of study.

Presentations:  On average, students presented 4 scientific papers at professional and scientific meetings.

Funded scholarships, fellowships, grants, etc.: On average, students were awarded 0 scholarships and fellowships, and received 2 grants for conducting research in their field of study.

Activities associated with professional associations:  The two Ph.D. students demonstrated active participation in professional societies. Both students professionalized into employment in their respective fields of study after completing their doctoral degrees.

Overall, graduate advisors should encourage graduate students to join and actively participate in professional societies and present their research at professional conferences. Because publishing in peer-reviewed journals is a lengthy process, evaluation of professionalization based on publication of research will require revised methodology to track students post-graduation.

Conclusions & Recommendations

The Department of Animal & Range Sciences awarded 17 M.S. and 3 Ph.D. degrees during January 2017 – August 2018.  The department met desired response thresholds for 4 of 5 learning outcomes identified in our Graduate Program Assessment Plan for students in our M.S. degree during the reporting period.  Two lines of evidence used to evaluate Learning Outcome 3 suggested we are close to meeting our targeted response threshold for effectiveness in written communication; evaluations completed by graduate student advisory committees indicated that 8 of 11 (73%) students evaluated met targeted standards in writing effectiveness, whereas independent evaluation of the writing assignments of a sample of M.S. students by the Graduate Assessment Committee indicated that 4 of 5 students (80%) met targeted standards.  

The department’s Graduate Program Assessment Plan mandates that a faculty response is required when an acceptable performance standard has not been met, and a strategy developed to improve areas where the performance standard has not been met. Possible responses include:

1. Gather additional data during the following review period to verify or refute the results.
2. Change something in the curriculum or program to try to improve performance.
3. Develop or select an alternative performance datum to assess outcome.
4. Change the acceptable performance threshold (must provide reasoning behind such a strategy).

The Graduate Assessment Committee recommends Option 1 as results were based on a relatively small sample of M.S. students.  Nevertheless, overall low scores in Learning Outcome 3 (effectiveness in written communication) relative to other Learning Outcomes suggest program improvements in the training of scientific writers.

Only three students completed Ph.D. degrees in our graduate program during 2017–2019.  Two of these students were included in this graduate program assessment; data were not available for one student who defended and graduated prior to the implementation of the departmental Graduate Assessment Plan in May 2018.  The department met desired response thresholds for 6 of 6 learning outcomes identified in our Graduate Program Assessment Plan for students in our Ph.D. degree during January 2018 – August 2019.

Appendix A.  Rubric for graduate assessment

1. Graduate advisors will complete the Thesis / Dissertation and Defense Assessment following an attempted graduate student defense.

4 = Exceeds Standards: Student demonstrates competent performance exceeding normal standards at either the M.S. or Ph.D. level.

3 = Meets Standards: Student demonstrates appropriate performance for professionalization

2 = Below Standards: Student does not demonstrate the skills commensurate with M.S. or Ph.D. degree.

1 = Unacceptable:  Performance is clearly inadequate. Student demonstrates an inability or unwillingness to develop appropriate skills.

Appendix B: Rubric for Assessment of Learning Outcome 3: Effectiveness in written communication of substantive content

4 = Exceeds Standards: Student demonstrates competent performance exceeding normal standards at either the M.S. or Ph.D. level.

3 = Meets Standards: Student demonstrates appropriate performance for professionalization

2 = Below Standards: Student does not demonstrate the skills commensurate with M.S. or Ph.D. degree.

1 = Unacceptable:  Performance is clearly inadequate. Student demonstrates an inability or unwillingness to develop appropriate skills.

Appendix C: Rubric for Assessment of: Effectiveness in oral communication of substantive content (Learning Outcome 4)

4 = Exceeds Standards: Student demonstrates competent performance exceeding normal standards at either the M.S. or Ph.D. level.

3 = Meets Standards: Student demonstrates appropriate performance for professionalization

2 = Below Standards: Student does not demonstrate the skills commensurate with M.S. or Ph.D. degree.

1 = Unacceptable:  Performance is clearly inadequate. Student demonstrates an inability or unwillingness to develop appropriate skills.

Click here for the pdf version of the 2018-2019 Graduate Students Assessment Results report.