How did the project start?
Before this project, our platform offered students the opportunity to rate each mentor session after it had taken place. This took the form of a single-screen pop-up that looked like this:
It gave students the opportunity to give the session a thumbs-up or a thumbs-down, and to add any extra comments. Feedback wasn’t shared with mentors by default, though students could opt in to this.
What we were finding, though, was that when students shared negative feedback on mentor sessions, they would usually share it only with us at Designlab. A couple of instances of negative feedback got me (Harish) and Paige, our Creative Director, talking.
We began this process by brainstorming over email. Here’s a bullet-point summary of our initial email exchange – we thought about turning it into a visual mindmap just for this article, but decided it would be more honest and insightful to be true to the medium through which we first discussed and got to grips with the problem. After all, this is the reality of how changes and improvements begin.
- Thumbs up? No
- Share with mentor? No
- Comments: My mentor had good advice but seemed distracted and somewhat disengaged. I got the sense that they wanted to rush through the session and finish up as soon as possible.
- See this student’s feedback – how do you think we can improve experiences like this?
- It’s hard for Designlab to mediate all student feedback about mentors
- Only scalable way to deliver feedback to mentors is going to be for it to be direct
- Needs to be pitched as constructive critique rather than a good or bad review
- Could ask questions like: - how was your session with your mentor? - what did your mentor do well? - what could your mentor improve? - any other comments?
- Share this automatically with mentors
- Don't shield mentors from negative feedback
- Mentors get unfiltered stream of both the good and the "could be better"?
- Encourage transparency
- Push mentors to improve over time
- Contribute to creating a culture of constructive critique and self-improvement
- Feedback the other way? Mentor feedback to students?
- e.g. challenge students to be more prepared in sessions, iterate on their work?
- Students don't always give feedback well
- Risk of unconstructive or hurtful language, which is counterproductive
- We can do a better job by providing scales with pre-determined language which forms a scalar rating system
- Don't want mentors to feel defensive or like their time isn't well spent
- Equipping students with better more compassionate language can improve the interaction and help both mentors and students to improve
- You’re right, it’s a delicate issue
- Easy to turn people off when the wrong language is used
- Extra sensitivity because mentors give up their time at a lower rate to benefit students
- If handled well, we can strengthen whole experience and platform
- If handled badly, it could alienate mentors or make specific student-mentor relationships worse rather than better
- As we scale the product need to set clear expectations for mentors
- Need to enforce student and mentor norms and expectations sooner rather than later
- Look to human resources materials for how to tune the language we use?
- Could begin with a start–stop–continue format
- Could be modeled on the surveys we give students about their learning from modules?
- We can nail this
In response to this initial brainstorm, we considered ways to improve our operational procedures around mentorship. We asked ourselves if we could we identify poor mentor experiences earlier – but that approach seemed incomplete.
As we thought more deeply about the problem, we were mindful that the bedrock of Designlab’s community is a culture of constructive feedback, mutual support, and personal growth. We believe that people excel in environments that foster transparency and trust. Just as students enroll in courses to improve their design skills, mentors also join Designlab so that they can grow as design leaders.
It quickly became clear that our existing feedback mechanisms were not adequately supporting these values and beliefs. We decided to investigate the tools we give students to provide feedback on their mentors.
Our next steps: questioning and researching
From these beginnings, as a team we undertook a process of questioning, researching, and reflecting on new learning. In particular, we sought a student’s-eye view on the problem. Consistent with the principles of design thinking, we sought to put ourselves in the user’s shoes (both student and mentor), to better understand their needs and goals when interacting via the Designlab platform.
We began to ask ourselves a lot of questions, including:
What could we improve or make available to students so that they can immediately report mentor problems? How can we help to turn negative experiences around?
What tools could we give to students to help them offer constructive critique to their mentors? How can we turn okay experiences into exceptional moments?
Could we extend our student community’s culture of trust and openness to critique? How can we offer mentors the benefit of constructive critique from their students?
How can we reward exceptional teachers and assist those experiencing difficulties? How can we fairly and reliably measure the quality of mentor performance?
Could we help teach mentors how to become better teachers?
We went on to identify five high-priority areas for research:
What does it mean to be a good mentor? We wanted to find out about existing, standardized definitions and expectations of mentorship. This would help us to define what good and bad mentoring means, both in terms of accepted norms, and specifically when it comes to the Designlab experience. We needed to identify a set of standards that would enable students to understand whether their own mentor’s performance was good or bad.
What feedback should be shared with mentors? We needed to establish what feedback should be shared with mentors. We wanted to be able to share unfiltered feedback with mentors, whether positive or negative. How should we go about educating students on what they can expect from mentors? And could we equip students with the skills to give critical feedback constructively and compassionately?
How should that feedback be shared? We researched how design teams and design leaders engage with designers in feedback and critique sessions. A key distinction that we would need to communicate to students is the distinction between constructive and non-constructive critique.
How can we ensure students feel comfortable giving feedback to mentors? We discovered that students can lack contextual understanding, and that can diminish their confidence when giving feedback. There can be a fear that “I don’t know enough” to offer critique. Also, students may come into courses with different levels of “soft skills”, meaning variable ability to use nuanced and carefully chosen language to engage constructively.
How can we ensure that the student-mentor relationship is happy and healthy? We needed to educate our mentors on what we expect from them. We researched mentor relationships in the workplace (like managers, coaches, and teachers). We compiled a list of mentorship best practices, and adapted these to fit a mentor-student model rather than a workplace model.
Based on this research, we began to draft some solutions to test ideas and uncover any hidden assumptions.
Draft attempts at a solution
Version 1: 5-star Ratings
“Version 1” took the existing design of the student-to-mentor feedback screen and made some changes — instead of just thumbs up and thumbs down, we tried out a 5 star rating system, which was followed by a free text comment box:
After drafting Version 1, we took a step back and asked ourselves:
What value does the 5 star rating system add? What does a 4-star experience represent vs. a 5-star experience? Do students have the language and skills to provide meaningful critique using a simple 5-star tool?
Will students feel safe and secure giving feedback in this way? Will students answer honestly? How much of a difference does it make to honesty when students know that their feedback will be shared with their mentor?
What kind of feedback is valuable to share with mentors? What should stay between the student and Designlab? How interventionist should Designlab be in facilitating the feedback cycle between mentors and students? Should it be more about creating a safe space for mentors and students to give each other feedback?
Version 2: Adding Performance Guidelines
Version 2 integrated a set of mentor performance guidelines that had come out of our research — making the 5-star rating feel much more robust, since students were given objective guidelines to review instead of simply rating the overall experience. Under each question, students had two comment fields to add feedback: one set of feedback would go directly to the mentor, and one set of feedback would be shared privately with the Designlab team.
Designlab’s performance guidance for mentors
Leave students timely feedback on their work: Mentors should leave comments online on student’s work submissions within the first 24-48 hours as expected.
Communicate about scheduling in a timely manner: Mentors should respond to student scheduling emails, including notifying students in advance if they needed to reschedule an upcoming session.
Give students specific feedback on their efforts: Mentors should state clearly what they observe, be specific and direct. Mentors should exhibit sincerity and whenever possible communicate face-to-face for both positive and negative performance efforts.
Coach student’s professional growth: Mentors should share words of wisdom that guide student behavior for success, ask students to contribute ideas to make improvements, and solve problems together.
Assess the situation: Mentors shouldn’t wait until things go wrong to talk about them. They should always discuss with students how things are going, review students overall performance, and adjust plans accordingly to keep priorities current.
Deliver skills training when needed: Mentors should give students step-by-step instruction and support that aids students in performing the required skills or procedures in a hands-on way.
Reinforce good performance: Mentors should notice and mention when students are doing quality work and demonstrating positive behaviors with the same effort that they noticed and mentioned when students are not doing their best work.
Ultimately, we decided that this approach (of students optionally sharing feedback with both Designlab and their mentor) was excessively complex and cumbersome — placing the burden on the student to decide which feedback to share with the mentor, and which to share with Designlab. As a result, we decided to share all feedback to mentors by default. If a student is having issues that they’re uncomfortable sharing with the mentor, they could raise it directly with the Designlab team via an “I’m having issues” form.
After sketching out this proposed solution, our main open question was whether students would feel comfortable sharing feedback with their mentors, in the case of any issues arising.
To gauge this, we conducted a series of informal interviews with students who had previously experienced problems with their mentors in a Designlab course. We displayed the wireframes, explained the mentor performance guidelines, and had in-depth conversations about whether students could have seen themselves using this feature to share their feedback.
In all cases, the response was affirmative: students would have loved to share their critique with mentors as soon as they had concerns. Based on these interviews, we felt comfortable moving forward with the feature as outlined.
The solution we shipped
Following experimentation with these drafts, we developed a Version 3, which represented an agreed solution. After every completed mentor session, students see a short mandatory survey on the site. They’re asked a series of 7 questions, each of which addresses one of Designlab’s mentor performance guidelines.
For each of these seven measures of performance, the student has the chance to share their feedback on a Likert scale of 1–5 (Strongly Disagree to Strongly Agree). They can also add extra comments that will be directly shared with their mentor.
The survey is lightweight enough to be completed quickly, but gathers enough information for it to be useful to the mentor, and to reinforce the importance of good communication and constructive feedback in the student–mentor relationship.
This is one of the 7 questions a student now sees after mentor sessions
If students are experiencing serious problems with a mentor that they’d rather share directly with Designlab, they can submit a short report to our support team.
On the mentor side, we’ve added those 7 performance guidelines to the onboarding process that takes place when a mentor signs up to mentor with Designlab. We are also providing mentors with more coaching on how to conduct effective sessions, and we’re encouraging mentors to proactively ask their students to share feedback with them.
With this system, our solution meets these key criteria from our outline:
Students and mentors have objective guidelines to judge the mentorship experience
Students have a forum to directly provide mentors with feedback to improve their relationships
Students can escalate more serious problems to the Designlab team
Designlab can observe mentor performance over time, and give mentors additional coaching and guidance
As with any release, we launch this feature with cautious optimism. We’re excited to see the results so that we can refine the solution further. As we track our data in the coming weeks, here are some of the questions we’ll be asking ourselves:
Even though we tried hard to make the feedback survey friendly and unintimidating, will students feel empowered to answer honestly?
Will mentors take student feedback seriously and use it to improve their sessions?
Is the survey too long?
Will the feedback shared by students enable mentors to act, or will it get repetitive?
And these are some of the measures we will use to assess the success of this product update:
Students feel like they can openly share critique with mentors
Mentors read and act on feedback from their students
Mentors’ skills improve over time
Mentors recognize Designlab as a place where they can not only get paid and mentor students, but also grow as people and as designers
We’re excited to see the impacts of the new ratings system on the Designlab platform and will let you know what we discover.