Project Overview and Purpose
Students in introductory biology courses often report they are studying hard yet not meeting their goals. Why do so many students struggle to learn despite their efforts? One likely reason, supported by work in the cognitive sciences, is that students are generally not adept at judging how well they know something (a metacognitive practice), nor at modifying their practices to improve their learning process (self-regulated learning; SRL) (Schraw et al., 2006; Stanton et al., 2015). Students who develop SRL strategies have better academic success and are more likely to persist in STEM courses than those who do not (Sebesta & Speth 2017; Hunter, 2019). SRL also strengthens students’ ability to transfer their knowledge to new settings and events (Bransford et al., 2000; Zhao et al., 2014), and developing such strategies can be especially impactful for students from historically underserved communities (e.g., Rodriguez et al., 2018). However, despite this evidence, few faculty encourage and engage students in SRL, and few resources exist to help create course structures that do so.
We propose to tackle this issue by developing new features for existing computational systems that encourage self-reflection and motivation through collecting and returning survey data to students while they are engaged in learning course materials. Students will be asked to answer questions about their performance goals and motivations, their self-regulatory strategies, and their confidence in their knowledge. Survey results will be returned to students, and followed by additional questions about how well they are currently meeting their goals, their likelihood of changing study strategies, and reflections on how they determine whether they are learning effectively. Students will also be messaged about upcoming assignments and quizzes, and offered choices of additional resources that include successful study practices, time management tips, etc. After each quiz/test in the course, they will be prompted to interface with the system again, always reflecting back on prior answers.
We propose that this technology-based intervention could have a profound effect on students by removing many of the barriers they often encounter. It should help them identify their personal motivations, set goals aligned with those motivations, and hone their study strategies through reflecting on their understanding and progress. In addition to developing the system, we will characterize the effects of using this system on students, measuring whether the process of repeated self-reflection and evaluation can lead to improvement in implementation of effective learning strategies, and in course performance.
We will begin the project by researching currently available online platforms that use surveys and analyses of responses to provide targeted feedback to participants, such as Neurowyzr (https://neurowyzr.com), MentorCoach (https://mentorcoach.com ), and ECoach (https://www.expertecoach.com/). ECoach, which is institutionalized in the Center for Academic Innovation at University of Michigan, is likely to have the most overlap with our needs as it is focused on educational environments and features such as messaging students and providing feedback before and after exams and providing space to make study plans and reflections. We will investigate whether Canvas and/or Qualtrics have sufficient capabilities to be customized to meet our project’s goals. We anticipate using these systems would lead to greater adoption of our designed intervention since students and educators would not need to install new software or learn how to interact with a new interface. Additionally, modifying established tools will help us to rapidly iterate on our designs and distribute our content more quickly. Our iterative design process will include soliciting feedback from both undergraduate students and faculty during the design and testing phases. We will also work with OIT and/or ASSETT at multiple points to ensure we understand capabilities of Canvas integration and/or integration of other software if deemed necessary.
Stakeholders and Intended Impact
Students (particularly in STEM) desperately need tools to help them be successful in their courses, and in understanding how to learn. In particular, students from under-served populations, first generation students, and transfer students often feel unwelcome in the university environment, and struggle to determine successful learning strategies. The proposed intervention could help many students at CU, in any discipline. Furthermore, since we will study the effects of this technology, the results will be disseminated widely, and have the potential to be taken up by others at a national level. This is even more likely if the designed tools are easy to integrate into a learning management system. Ultimately, this technology could help promote more equitable learning environments that can reduce opportunity gaps.
Team Description/Gaps
Dr. Knight is a biology educational researcher and has the tools to qualitatively analyze student responses and create statistical models for what types of behaviors promote success. The team also includes MCDB PhD student Zachary Hazlett, undergraduate MCDB major Justin Hein, and Dr. Jason Zietz, Assistant Teaching Professor, Department of Information Science. Dr. Zietz has the computer skills and cognitive science background to help design and iterate on the computational interventions, and will identify one or more skilled undergraduate students who will aid with computer programming. We will also use additional undergraduate students for testing the system. Hazlett will lead the exploration of current technologies, including attending a summer conference that focuses on ECoach (https://www.seismicproject.org/annual-meetings/2023-summer-meeting/) to learn how that system works.
Intended Scale
Initially, students in one or two courses at CU would be asked to engage in this project, beginning with MCDB’s large Genetics course (~200 students/section, taught in Spring by Knight). Zietz’s courses in computer programming are other likely test cases. Once the system is designed and tested, we can implement it in other courses at CU (e.g. ones with typically high DFW rates such as Introductory Biology and Chemistry), at the discretion of interested faculty, and ultimately it could be incorporated in any course at CU or elsewhere.
Funding Request + Intended use of Funds: total $63,057
- Graduate student Zachary Hazlett will investigate current technology, help plan and guide development of the product, work with Dr. Zietz to troubleshoot the system, and interface with students to determine its efficacy: (25% appointment, 4 semesters): $32,224
- Undergraduate student assistant: Justin Hein, junior MCDB major with experience in education (Learning Assistant, Upward Bound instructor). Justin will be involved in development and testing of the product, as well as interfacing with additional students for feedback. $20/hr, ~75 hours = $1,500.
- Undergraduate student assistant (TBD) identified from Dr. Zietz’s programming courses. $20/hr, ~75 hours = $1,500.
- Undergraduate students who have recently taken or are currently taking Genetics: ~20 @$20 each: students will be interviewed to help characterize their experience using the product = $400.
- Assistant Teaching Professor Dr. Jason Zietz: Two months of summer salary over two years: $21,433
- Associate Professor Jenny Knight: $3,000 per summer over two years to coordinate the program: $6,000
Anticipated Long Term Needs
If the system works well using off-the-shelf technology, minimal long-term support would be needed for running the system. We would interface with ASSETT to make the system attractive to other users and offer workshops through ASSETT and/or CTL to encourage faculty adoption.
References
Bransford, J. D., Brown, A. L., Cocking, R. R. (2000). How People Learn: Brain, Mind, Experience, and School: Expanded Edition. In How People Learn: Brain, Mind, Experience, and School. https://doi.org/10.17226/9853
Hunter, A.-B. (2019). Why undergraduates leave STEM majors: Changes over the last two decades. in Talking about Leaving Revisited: Persistence Relocation and Loss in Undergraduate STEM Education, Cham, Switzerland:Springer, pp. 87-114, 2019. https://doi.org/10.1007/978-3-030-25304-2
Rodriguez, F., Rivas, M. J., Matsumura, L. H., Warschauer, M., Sato, B. K. (2018). How do students study in STEM courses? Findings from a light-touch intervention and its relevance for underrepresented students. PLoS ONE, 13(7), 1–20. https://doi.org/10.1371/journal.pone.0200767
Schraw, G., Crippen, K. J., Hartley, K. (2006). Promoting self-regulation in science education: Metacognition as part of a broader perspective on learning. Research in Science Education, 36(1–2), 111–139. https://doi.org/10.1007/s11165-005-3917-8
Sebesta, A. J., Speth, E. B. (2017). How should I study for the exam? Self-regulated learning strategies and achievement in introductory biology. CBE Life Sci. Educ. 16(2), 1–12. https://doi.org/10.1187/cbe.16-09-0269
Stanton, J. D., Neider, X. N., Gallegos, I. J., & Clark, N. C. (2015). Differences in Metacognitive Regulation in Introductory Biology Students: When Prompts Are Not Enough. CBE Life Sci. Educ., 14(2), ar15–ar15. https://doi.org/10.1187/cbe.14-08-0135
Zhao, N., Wardeska, J. G., Mcguire, S. Y., Cook, E. (2014). Metacognition: An Effective Tool to Promote Success in College Science Learning. Journal of College Science Teaching, 43(4), 48–54.
Return to the ASSETT Webpage
This looks amazing! If you are looking for student involvement, we could propose this to our ASSETT students who serve on the advisory committee– even if just as advisors to your proposal.
that would be great-i’d welcome student participation and input
Hi Jenny- Your initial proposal is exciting and builds on a past project of the innovation incubator. Take a look at this Canvas exam reflection module developed by Becca Ciancanelli and Shane Oshetski. I would be interested in talking with you about whether this tool could be integrated into your proposed project in any way. https://fedauth.colorado.edu/idp/profile/SAML2/Redirect/SSO?execution=e1s1
I will also work with Beth to follow-up on engaging ASSETT student advisors in providing feedback on your idea.
This is a great project idea and I am so excited to see the quality of research you have already put into it. I have a few technical suggestions for your proposal which might help you think through what technology you may need/use to create this service:
1. Google App Scripts [https://www.google.com/script/start/]: Through Google you can create apps that integrate with Google Forms, Sheets, or other Google Products. I think it is very doable to have a student worker create a Google Form designed to support the SRL and write a Google Script to process the students responses and provide feedback in some form. Also, since this would be a Google Form, the output could be collected in a Google Sheet which may remove the need to maintain a web server and/or database.
2. Microsoft Power Apps [https://powerapps.microsoft.com/en-us/] CU currently provides access to Microsoft 365 apps/programs, which includes Power Apps. With a Power App you can create a app/website that students can interact with and then process their data via Microsoft’s servers. I do not know all the limitations of our current licensing, but this could reduce or even eliminate your hosting costs.
3. Graduate Student Work Status – You have set aside funding for graduate student workers, but I would recommend listing whether you plan to hire them as hourly workers or with a 50% work appointment. According to the Graduate School a 50% appointment costs $12,084 per semester, so a full year would be closer to $24,000 rather than the listed $15,000. So add a few more words on what type of employment classification you are considering (appointment vs hourly) [https://www.colorado.edu/graduateschool/funding/departmental-funding/graduate-student-appointment-information]
This is a great draft with a strong, well-researched motivation. I especially appreciate the effort to develop a procedure or tool to help practically implement metacognitive skills. I can see implementing such a process/intervention in my own classes. My only suggestion–and I’m providing this feedback to most of the proposals–is that you might do a bit more to specify how the “design thinking” process the incubator facilitates will help you refine your idea and will help you develop a sustainability program. I could see this proposal potentially attracting funding; has would CTL, RIO, and/or OIT collaboration help you develop your idea and achieve the kind of impact at CU and beyond that you project.
This is a great proposal, Jenny! Using survey data to encourage self-reflection and motivation is a growing trend in education technology, and there are platforms and systems available that can help facilitate this process. I wonder if as part of this project, you could include a research of companies working on this area that you can partner with to solve this particular challenge. For example, MentorCoach is a coaching and mentoring platform that uses surveys to collect data from students about their progress and learning goals. The platform then uses this data to provide personalized coaching and mentoring to help students achieve their goals. Another company that may be interesting to explore is called “Cognifyx”. Cognifyx is an online platform that collects data from students through surveys and uses the data to provide personalized feedback to students. The platform uses a combination of cognitive psychology and artificial intelligence to analyze student responses and provide feedback that can help students improve their learning outcomes. I am sure there are other systems available. Maintaining such platforms secure, accessible, and updated require large investments; for the long term success of this project, it may make sense to be a developer partner of an established company. Good luck!
Hello, congratulations on the submission of your proposal. Firstly, I’d like to appreciate the fact that a student’s problems are being addressed and attended in a much more efficient manner. The project is looking beyond what is right and wrong and is exploring the a student’s circumstance and trying to better it by the use of metacognitive practices. As a student, it is exciting to know that a student’s circumstances are being taken into consideration and there are actions that can be taken to help students using self-reflection.