Department of Animal and Range Sciences

by Dr. Lance McNew and Dr. Sarah McCoski

The Department of Animal and Range Sciences (hereafter “department”) offers a Master of Science degree in Animal & Range Sciences and a Doctor of Philosophy degree in Animal & Range Sciences. Both the M.S. and Ph.D. degrees require that the student choose either an Animal Science emphasis or a Range Science emphasis.  In addition, Ph.D. students may opt to enroll in the interdisciplinary Ecology and Environmental Sciences degree program. Ph.D. students enrolled in this degree program are affiliated with a home department that corresponds to that of their major faculty advisor, and are required to meet standard departmental requirements.

Department faculty met during the Spring of 2018 to formulate a plan to assess the department’s graduate program.  A final plan was approved in May of 2018 and immediately implemented.  As required by the plan, the department is required to report on assessment activities and critically evaluate the department’s graduate program.  The following documents serve as our assessment for the time period January 2018 – August 2019.

The Department identified the following desired learning outcomes for graduate students completing Master of Science and Ph.D. degrees:

For M.S. students:

  1. Conduct research resulting in an original thesis or dissertation
  2. Demonstrate mastery of subject content knowledge and research/critical inquiry methodology
  3. Demonstrate effective written communication of substantive content
  4. Demonstrate effective oral communication of substantive content
  5. Be able to conduct scholarly or professional activities in an ethical manner

For Ph.D. students:

  1. Produce and defend an original significant contribution to knowledge
  2. Demonstrate mastery of subject content knowledge and research/critical inquiry methodology
  3. Demonstrate excellence in written communication of substantive content
  4. Demonstrate excellence in oral communication of substantive content
  5. Be able to conduct scholarly and professional activities in an ethical manner
  6. Demonstrate professionalization into the field of study as demonstrated through publications, presentations, funded fellowships, professional association activities, professional experience, etc.

 

To evaluate departmental success in meeting desired learning outcomes, we collected student data from the following sources:

Student Performance: Data Sources

M.S. Programs

  Outcomes
Data Source 1 2 3 4 5
Departmental Records1 x        
Thesis/thesis defense2   x x x x
ARNR 507: Research Methods     x x  
Ethics training in responsible conduct of research3         x

PhD. Programs

  Outcomes
Data Source 1 2 3 4 5 6
Departmental Records1 x          
Qualifying exam   x x      
Dissertation/dissertation defense2    x x x x  
ARNR 507: Research Methods      x x    
Ethics training in responsible conduct of research3         x  
C.V.4           x

1 Program will track thesis/dissertation defenses and calculate success rates. 2 The rubrics for outcomes 1–4 are to be completed at the thesis/dissertation defense. If departments prefer, they may use comprehensive exams rather than the actual dissertation, thesis, or paper.  These rubrics will not be used to assess or evaluate individual students and will not inform the decision regarding whether a student passes a defense or course.  The data will be aggregated for all students in the program over a two-year period in order to assess the success of the program in meeting its program learning outcomes (see sample rubric attached).  3 The rubric for outcome 5 is the completion of some ethics training.  4 C.V.s of students will be collected as they complete their degrees.  The C.V. will be used to provide evidence of appropriate professional activities to the particular field of study (these may be published papers, research training, teaching development, presentations at conferences, etc. as is appropriate for your discipline or program and as was identified in learning outcome 6 above).

Results:

During January 2017 ­– August 2019, 17 M.S. and 3 Ph.D. students attempted to defend graduate degrees.  All graduate students were successful. Five M.S. students withdrew from the graduate program during the reporting period. Six Ph.D. students successfully completed comprehensive exams.

M.S. Students

Learning Outcome 1: Conduct research resulting in an original thesis or dissertation.

During January 2017 – August 2019, 17 students attempted to defend an M.S. thesis describing original research; all 17 produced an original thesis and successfully defended.  Graduate students successfully defending theses and earning M.S. degrees during January 2017 – August 2017 are listed in Table 1.

Learning Outcome 2: Demonstrate mastery of subject content knowledge and research/critical inquiry methodology

Beginning in 2018, graduate student advisory committees completed a Thesis and Defense Assessment (Appendix A), where each member of the graduate supervisory committee ranked the student’s performance relative to the learning outcome from 1 (Unacceptable) to 4 (Exceeds standards).  Scores were tabulated and averages calculated for each student.  Assessments were completed for eleven M.S. students defending during January 2018­—August 2019. 

The mean (± SD) score relative to Learning Outcome 2 was 3.4 (0.6).  Ten (10) of 11 students received average scores ≥3, indicating that 91% of the M.S. students defending during January 2018 – August 2019 met departmental standards relative to the learning outcome.  Thus, we exceeded the departmental response threshold that ≥ 80% of M.S. students will be ranked at a 3–4 level in subject content knowledge.

Learning Outcome 3: Demonstrate effective written communication of substantive content

Learning Outcome 3 was assessed using two data sources. First, graduate student supervisory committees completed a Thesis and Defense Assessment (Appendix A), where each member of the graduate supervisory committee ranked the student’s performance relative to the learning outcome from 1 (Unacceptable) to 4 (Exceeds standards).  The second data set in composed of writing assignments provided by graduate students as part of ARNR 507: Research Methods.  Each semester of course offering, two members of the faculty who were not the primary instructor for the course scored a random sample of writing assignments from students taking the class for the second or third time (i.e., near the end of their graduate program). We used the rubric presented in Appendix B to evaluate students relative to the learning outcome.

The mean (± SD) score collected by graduate committees at defenses relative to Learning Outcome 3 was 3.1 (0.5).  Eight (8) of 11 students received average scores of ≥3, indicating that 73% of the M.S. students met departmental standards relative to the learning outcome.  During January–August 2018, we did not meet our response threshold that ≥ 80% of students will be ranked at a 3–4 level in written communication. 

The Graduate Assessment Committee independently evaluated the writing assignments of five M.S. students enrolled in their second semester of ARNR 507.  Students were ranked based on four criteria and average scores (e.g., 1 = unacceptable, 4 = exceeds standards) across criteria calculated for each student.  Average writing scores for M.S. students was 3.1 (0.5).  Four of 5 students evaluated had overall scores ≥3, indicating that 80% of students sampled met departmental standards for effective written communication.

Learning Outcome 4: Demonstrate effective oral communication of substantive content

Learning Outcome 4 was assessed using two data sources. First, graduate student supervisory committees completed a Thesis and Defense Assessment (Appendix A), where each member of the graduate supervisory committee ranked the student’s performance relative to the learning outcome from 1 (Unacceptable) to 4 (Exceeds standards).  The second data set in composed of independent evaluations of oral presentations given by graduate students as part of ARNR 507: Research Methods.  Each semester of course offering, two members of the faculty who were not the primary instructor for the course scored a random sample of writing assignments from students taking the class for the second or third time (i.e., near the end of their graduate program). We used the rubric presented in Appendix C to evaluate students relative to the learning outcome.

The mean (± SD) score collected by graduate advisory committees at M.S. defenses relative to Learning Outcome 4 was 3.5 (0.5).  All 11 M.S. students defending during the reporting period received average scores of ≥3, indicating that 100% of the M.S. students met departmental standards relative to proficiency in oral communication. 

The Graduate Assessment Committee independently evaluated the oral presentations of 7 M.S. students enrolled in ARNR 507 during the January 2018 – May 2019). Average (SD) scores for effectiveness in the oral communication of student research was 3.6 (0.3).  All 7 students (100%) students received overall oral presentation scores ≥3.0 (range = 3.1–3.9).

Overall, both lines of evidence indicate that the department met the desired response threshold of M.S. student proficiency in oral communication.

Learning Outcome 5: Be able to conduct scholarly or professional activities in an ethical manner

All students received training in scientific ethics as part of ARNR 507.  In addition, graduate student supervisory committees completed a Thesis and Defense Assessment (Appendix A), where each member of the graduate supervisory committee ranked the student’s performance relative to the learning outcome from 1 (Unacceptable) to 4 (Exceeds standards).

The mean (± SD) score collected by graduate committees at defenses relative to Learning Outcome 5 was 3.7 (0.5).  All 11 students defending during the reporting period received average scores of ≥ 3, indicating that 100% of the M.S. students defending during 2018—2019 met departmental standards relative to ethical scholarly and professional activities.

 

Ph.D. Students

Learning Outcome 1: Produce and defend an original significant contribution to knowledge

During 2017–2018, three students attempted to defend a Ph.D. thesis describing original research, all of which (100%) produced an original dissertation and successfully defended. Graduate students successfully defending dissertations and earning doctoral degrees during January 2017 – August 2017 were Omolola Betiku, Neto Garcia, and Samuel Wyffels (Table 1).

Learning Outcome 2: Demonstrate mastery of subject content knowledge and research/critical inquiry methodology

Beginning in 2018, graduate student advisory committees completed a Thesis and Defense Assessment (Appendix A), where each member of the graduate supervisory committee ranked the student’s performance relative to the learning outcome from 1 (Unacceptable) to 4 (Exceeds standards).  Scores were tabulated and averages calculated for each student.  Assessments were completed for two Ph.D. students defending during January 2018­—August 2019. 

The mean (± SD) score relative to Learning Outcome 2 was 3.4 (0.6).  Both students received average scores ≥3, indicating that 100% of the doctoral students defending during January 2018 – August 2019 met departmental standards relative to the learning outcome.  Thus, we met our departmental response threshold that ≥ 80% of students will be ranked at a 3–4 level in subject content knowledge. 

Learning Outcome 3: Demonstrate effective written communication of substantive content

Learning Outcome 3 was assessed using two data sources. First, graduate student supervisory committees completed a Thesis and Defense Assessment (Appendix A), where each member of the graduate supervisory committee ranked the student’s performance relative to the learning outcome from 1 (Unacceptable) to 4 (Exceeds standards).  The second data set in composed of writing assignments provided by graduate students as part of ARNR 507: Research Methods.  Each semester of course offering, two members of the faculty who were not the primary instructor for the course scored a random sample of writing assignments from students taking the class for the second or third time (i.e., near the end of their graduate program). We used the rubric presented in Appendix B to evaluate students relative to the learning outcome.

The mean (± SD) score collected by graduate committees at defenses relative to Learning Outcome 3 was 3.3 (0.3).  Both Ph.D. students (100%) defending during 2018-2019 received average scores of ≥3.

The Graduate Assessment Committee independently evaluated the writing assignments of three Ph.D. students enrolled in their second semester of ARNR 507.  Students were ranked based on four criteria and average scores (e.g., 1 = unacceptable, 4 = exceeds standards) across criteria calculated for each student.  Average writing scores for Ph.D. students was 3.8 (0.3).  All three students evaluated had scores ≥3, indicating that 100% of students met departmental standards for effective written communication.

Learning Outcome 4: Demonstrate effective oral communication of substantive content

Learning Outcome 4 was assessed using two data sources. First, graduate student supervisory committees completed a Thesis and Defense Assessment (Appendix A), where each member of the graduate supervisory committee ranked the student’s performance relative to the learning outcome from 1 (Unacceptable) to 4 (Exceeds standards).  The second data set in composed of independent evaluations of oral presentations given by graduate students as part of ARNR 507: Research Methods.  Each semester of course offering, two members of the faculty who were not the primary instructor for the course scored a random sample of writing assignments from students taking the class for the second or third time (i.e., near the end of their graduate program). We used the rubric presented in Appendix C to evaluate students relative to the learning outcome.

The mean (± SD) score collected by graduate committees at defenses relative to Learning Outcome 4 was 3.4 (0.4).  Both students defending during the reporting period received average scores of ≥3, indicating that 100% of the Ph.D. students met departmental standards relative to proficiency in oral communication based on graduate committee assessment. 

During three semesters of ARNR 507 (spring 2018 & 2019, fall 2019), we evaluated the oral presentations of two Ph.D. students.  Average scores for effectiveness in the oral communication of student research was 3.4 (0.4).  Both (100%) students received overall average scores ≥3.0.

Overall, both lines of evidence indicate that the department met the desired response threshold of Ph.D. student proficiency in oral communication.

Learning Outcome 5: Be able to conduct scholarly or professional activities in an ethical manner

All students received training in scientific ethics as part of ARNR 507.  In addition, graduate student supervisory committees completed a Thesis and Defense Assessment (Appendix A), where each member of the graduate supervisory committee ranked the student’s performance relative to the learning outcome from 1 (Unacceptable) to 4 (Exceeds standards).

The mean (± SD) score collected by graduate committees at defenses relative to Learning Outcome 5 was 3.7 (0.1).  Both students defending during the reporting period received average scores of ≥ 3, indicating that 100% of the Ph.D. students defending during 2018—2019 met departmental standards relative to ethical scholarly and professional activities.

Learning Outcome 6: Demonstrate professionalization into the field of study as demonstrated through publications, presentations, funded fellowships, professional association activities, professional experience, etc.

The curriculum vitae of two Ph.D. students were collected after their doctoral defense and evaluated for evidence of professionalization into their field of study. 

Publications: On average, Ph.D. students published 1 publication in peer-reviewed and 5 other scientific/professional journals associated with their field of study.

Presentations:  On average, students presented 4 scientific papers at professional and scientific meetings.

Funded scholarships, fellowships, grants, etc.: On average, students were awarded 0 scholarships and fellowships, and received 2 grants for conducting research in their field of study.

Activities associated with professional associations:  The two Ph.D. students demonstrated active participation in professional societies. Both students professionalized into employment in their respective fields of study after completing their doctoral degrees.

Overall, graduate advisors should encourage graduate students to join and actively participate in professional societies and present their research at professional conferences. Because publishing in peer-reviewed journals is a lengthy process, evaluation of professionalization based on publication of research will require revised methodology to track students post-graduation.

Conclusions & Recommendations

The Department of Animal & Range Sciences awarded 17 M.S. and 3 Ph.D. degrees during January 2017 – August 2018.  The department met desired response thresholds for 4 of 5 learning outcomes identified in our Graduate Program Assessment Plan for students in our M.S. degree during the reporting period.  Two lines of evidence used to evaluate Learning Outcome 3 suggested we are close to meeting our targeted response threshold for effectiveness in written communication; evaluations completed by graduate student advisory committees indicated that 8 of 11 (73%) students evaluated met targeted standards in writing effectiveness, whereas independent evaluation of the writing assignments of a sample of M.S. students by the Graduate Assessment Committee indicated that 4 of 5 students (80%) met targeted standards.  

The department’s Graduate Program Assessment Plan mandates that a faculty response is required when an acceptable performance standard has not been met, and a strategy developed to improve areas where the performance standard has not been met. Possible responses include:

  1. Gather additional data during the following review period to verify or refute the results.
  2. Change something in the curriculum or program to try to improve performance.
  3. Develop or select an alternative performance datum to assess outcome.
  4. Change the acceptable performance threshold (must provide reasoning behind such a strategy).

The Graduate Assessment Committee recommends Option 1 as results were based on a relatively small sample of M.S. students.  Nevertheless, overall low scores in Learning Outcome 3 (effectiveness in written communication) relative to other Learning Outcomes suggest program improvements in the training of scientific writers.

Only three students completed Ph.D. degrees in our graduate program during 2017–2019.  Two of these students were included in this graduate program assessment; data were not available for one student who defended and graduated prior to the implementation of the departmental Graduate Assessment Plan in May 2018.  The department met desired response thresholds for 6 of 6 learning outcomes identified in our Graduate Program Assessment Plan for students in our Ph.D. degree during January 2018 – August 2019.

Table 1. List of graduate students enrolled in the Department of Animal & Range Sciences, Montana State University
Year Name Degree Pass/Fail/Withdrew
2017

Elkins, Eric

Herryger, Melissa

Betiku, Omolola

Cover, Cayce

Staudenmeyer, Danielle

Wells, Smith

Hubbard, Ashton

McGregor, Ian

Page, Chad

Blatter, Sarah

Payne, Jarrett

Ritter, Tory

Sapkota, Anish

Vold, Skyler

MS

MS

PhD

MS

MS

MS

MS

MS

MS

MS

MS

MS

MS

MS

Pass

Pass

Pass

Withdrew

Pass

Pass

Pass

Pass

Pass

Pass

Pass

Pass

Pass

Pass

2018

Johnson, Brittani

Garcia, Neto

Williams, Alyson

Arias, Carlos

Mueller, Anna Luka

Knuth, Ryan

Pulliam, John

Kurzen, Mark

MS

PhD

MS

MS

MS

MS

MS

MS

Pass

Pass

Pass

Withdrew

Withdrew

Pass

Pass

Pass

2019

Wyffels, Sam

Bauer, Rory

Chulyak, Victoria

PhD

MS

MS

Pass

Withdrew

Withdrew

Comprehensive Exams
2016 Betiku, Omolola PhD Pass
2017
Garcia, Neto
Wyfells, Samuel
PhD Pass
2018
Milligan, Megan
Hieber, Jordan
PhD Pass
2019 Eiseman, Krista PhD Pass

 

Appendix A.  Rubric for graduate assessment

  1. Graduate advisors will complete the Thesis / Dissertation and Defense Assessment following an attempted graduate student defense.

4 = Exceeds Standards: Student demonstrates competent performance exceeding normal standards at either the M.S. or Ph.D. level.

3 = Meets Standards: Student demonstrates appropriate performance for professionalization

2 = Below Standards: Student does not demonstrate the skills commensurate with M.S. or Ph.D. degree.

1 = Unacceptable:  Performance is clearly inadequate. Student demonstrates an inability or unwillingness to develop appropriate skills.

Thesis / Dissertation and Defense Assessment

(score each component 1-4)

Score
Demonstrates mastery of subject content knowledge and research/critical inquiry methodology  
Demonstrates effective written communication of their study  
Demonstrates effective oral presentation of their study  
Conducted study and professional activities during their program in an ethical manner  
Average score  

Appendix B: Rubric for Assessment of Learning Outcome 3: Effectiveness in written communication of substantive content

4 = Exceeds Standards: Student demonstrates competent performance exceeding normal standards at either the M.S. or Ph.D. level.

3 = Meets Standards: Student demonstrates appropriate performance for professionalization

2 = Below Standards: Student does not demonstrate the skills commensurate with M.S. or Ph.D. degree.

1 = Unacceptable:  Performance is clearly inadequate. Student demonstrates an inability or unwillingness to develop appropriate skills.

Indicators of Effective Written Communication of Substantive Content 1 2 3 4 Score
Style / Organization Paper is poorly written and reveals a lack of effort suitable for a graduate student Paper conveys appropriate ideas, but reveals weak control over diction, syntax, and organization Effective command of sentence structure and diction.  Paper is organized in a logical scientific manner Excellent command of sentence structure, diction, and organization is appropriate for subject matter content  
Content Major omissions necessary for scientific paper Some necessary components of an effective paper missing or poorly described Good job presenting ideas; contains all necessary content for scientific paper, but not as clear or succinct as it could be Clearly presents appropriate justification, objectives and methods; If available, results are complete and inferences follow from the data  
Grammar Weak grammar, spelling Several grammar and spelling errors Few spelling and grammar errors No spelling or grammar mistakes  
Sources Poorly sourced Some major relative literature not covered Major relative literature discussed Exhaustive literature presented  

Appendix C: Rubric for Assessment of: Effectiveness in oral communication of substantive content (Learning Outcome 4)

4 = Exceeds Standards: Student demonstrates competent performance exceeding normal standards at either the M.S. or Ph.D. level.

3 = Meets Standards: Student demonstrates appropriate performance for professionalization

2 = Below Standards: Student does not demonstrate the skills commensurate with M.S. or Ph.D. degree.

1 = Unacceptable:  Performance is clearly inadequate. Student demonstrates an inability or unwillingness to develop appropriate skills.

 

Indicators of Effective Oral Communication of Substantive Content 1 2 3 4 Score
Organization Poor Insufficient Adequate Presentation is arranged logically  
Content Omission of critical information necessary for a scientific presentation Missing key components of effective presentation Most components covered, but talk would benefit from additional information Material presented was complete and appropriate, all key components covered  
Clarity Study justification, objectives, and methods unclear; demonstrated lack of preparation Slides poorly arranged or improperly formatted.  Font size too small, too crowded, inappropriate color scheme, overuse of acronyms and jargon Presentation is relatively clear; some slides too busy or lacking; visual aids are well designed, legible, with appropriate content Presentation is succinct and clear; avoids jargon and acronyms; visual aids are well designed, legible, with appropriate content  
Knowledge & Understanding Demonstrates poor knowledge of the materials presented Demonstrates a lack of knowledge in critical components of the study (e.g., literature, study design, analyses) Demonstrates solid understanding of the topic and associated literature; highlights important points where study is strongest; delivers effective conclusion Demonstrates a superb grasp of the topic and the literature related to the topic; well prepared for questions; Revisits important and relative points  
Delivery Obvious ill-preparedness Ineffective delivery; poor speech mechanics; nervous habits interfered with effective presentation Effective delivery; appropriate volume, few nervous habits, relatively little reliance on notes; evidence of preparation Outstanding delivery; engagement with audience, little reliance on notes, smooth transitions  

 

Click here for the pdf version of the 2018-2019 Graduate Students Assessment Results report.