Case study: Optimizing churn for a product at scale - with Mathias Kluge

Welcome everyone to another episode of the Product Bakery. I'm sitting here today with Christian and Matze. We are in person today. I had my negative corona test yesterday evening and I actually had one this morning as well but I still don't have the result. But I think we are in a safe place to have this conversation. With enough distance. Yes, correct. Before we jump to the conversation I wanted to point out that you can find all our information on the website if you go to product-bakery.com. There is also a page where you can find all the episodes with some episode minutes as well as a transcript of the first couple of questions and you have the possibility to add your comments. So if you have any questions, if you have feedback or if you just want to shout out to our speakers you can use the comment functionality as well as our social media channels. So we're looking forward to hearing from you and with this I hand it over to Christian. Thanks Alex. So we have recently received a couple of emails to go a little bit deeper into one or the other topic and we thought today would be a good day to start with a case study and we thought there couldn't be a better person than Matthias Kluge who is a product lead at Thread Republic at the moment to pick one of his previous projects and make a deep dive to share the best practices and how to set up projects. But before I talk about this, Matze, you have originally started your career as a product designer and you worked your way up into the world of product management and now you are a product lead at Thread Republic. Sounds like he's the perfect candidate for product bakery, right? Like combining different roles, becoming a product manager as a product designer. I guess you have some nice stories to tell. He is, I think, the mix of both of us. Merge. But as always, first of all, how do you feel today? Great, boy oh boy, what a year. Recommended for future generations. Thanks for having me. Yeah, that's some expectations to live up with, yeah. So I'm probably the unmarried child of both of you, as it looks like. Happy to be here. I'm currently working, as you said, as the product lead of Thread Republic. I dabbled a few years in product design, moved into, I wouldn't say up, like across or sideways into product management and yeah, gathered some experience and maybe I can share some stuff here. Are you still enjoying product management? Are you missing product design? I go on the nerves of my designers all the time to satisfy or scratch my itch around product design. Sometimes they like it, sometimes they don't like it. But no, I don't miss it. As long as I have designers around me that I can talk to, that I can challenge or that can render whatever I say to them stupid, that's fine. So at least there is, as long as there is an exchange going on design, I'm quite happy. I think we definitely touch up on the topic of working with designers today, but we thought the perfect topic is one of the projects you did in the past and it was called optimizing churn for product at scale. So that was the topic you choose and I would love to hear from you what it was and how to get started with it. Yeah, I think it's a great thing because every product person in their career will either work on this, experience this or see some of their colleagues experiencing this. So basically at some point you founded a startup, your product is successful and sooner or later you look at your churn, which basically means the customers that are leaving your product to decide and then you want to figure out ways how to optimize for that. So the question is how to approach this topic when you're tasked with it, what to measure first and yeah. Let's imagine we are selling a product, let's say we are an e-commerce company and we just have a couple of customers who are loving us but also realize okay they would like to go somewhere else. So what would you recommend doing us or how have you done it in the past? First and foremost when you're a product manager in this situation is you have to find out where you're at. So that's generally the major struggle and major iterative journey of product management to find out where you're at in the stage of let's call it the method or the process. Are you in an implementation phase? Are you in a discovery or are you trying to prove a hypothesis? What all these words mean, it's interchangeable. So the question is you observed something. You have data or you don't have data. The first thing you need to know is basically you need success metrics and these success metrics need to be based on facts. Otherwise everything you do you will not be able to argue or justify any sort of outcome of it. So that's the first point. So don't get moving right away and brainstorming what could be possible solutions to stop customers from churning. Many examples like solving problems they have, giving them an upsell, giving them a voucher, whatever to not leave your platform. Is there a particular example you can tell us? Let's say a particular example from previous experience. We had a product that customers used on a day-to-day basis and what we defined as churn is basically customers stopping using it. But what you only see is a result. You don't know why. The first point is you need to design an experiment or design something that gives you data. Otherwise you cannot act. You're operating on assumptions. When you operate on assumptions you cannot really measure each of the improvements that you do if they have effect and why this effect was there. So point number one is to create a proper setup for the data you want to collect. In which data did you look for your particular case? The first thing is asking the customers why they're going to churn or why they leave your product for example. In the case of a churn was it like people canceling their accounts or did they stop using it and you just saw that the traffic is not there anymore? Okay yeah let's go a little bit specific. Let's start with the case when customers canceling their account. When you have an account based products and the customers canceling their account what's the logical conclusion which seems obvious and simple is when customers cancel their account ask them why they do it. So what I see very often canceling accounts on my own that you usually have to go through an extensive journey of questions. Some of them are mandatory. So even the process of canceling the account for the user feels cumbersome. It feels like they want to do everything to keep you in the platform or scare you even away from doing it and that gives a way worse connotation to the brand. If you lose a customer like that you will never convert them back because they're already frightened and they're just happy to be out of this. So usually in my experience it comes down to two sets of data that you want. It's either data about how satisfied they are with their product or data about usage of their product. How this plays out in detail it depends on your product what you actually want to ask the users but limit it to two to three simple selectable let's call it pills when you want to talk in UI elements. Don't make them go through form or something. Really think about what is the minimum data set that you need that gives you enough indication of a specific improvement. And now I can talk about a mistake that I made early on which taught me a lot about thinking ahead about data you collect. So the idea that I had was basically hey let's ask them are you satisfied with the product or not or basically are you not satisfied. Did you stop using it. That was the data I was interested in. And also you can leave us a comment. Yeah. So they could also leave us a free form comment and I thought that would be great. Then I can really dig into it and find out what the real problem and struggle of the customers are ignoring the fact that I was quite large product with a large amount of customers and getting 3000 comments every week became very quickly a problem on its own to handle. So it taught me very direct and with a lot of pain that you need to think ahead what you want to do with your data. I mean isn't that something where you can just like say hey designers researchers go through the data figure it out. Yeah you could do that if you want to destroy the little bit of relationship you have left with them. But you have two options basically. You either take a resource an engineering resource and look at natural language processing of this data which is a cool project on that on its own for a hackathon or something you really want to invest time in. But still if you don't plan for it you have another ticket setting you have another feature setting in there and this goes in the realm of machine learning and gets quite complex early on so it's fuzzy. And the other option is to hire a bunch of interns especially that can do that for you. But in the end the lesson learned here is be careful with the data you asked for because someone has to implement it and someone has to analyze it. If it's not analyzable it sits there unused basically. Or you take three hours go through the spreadsheet cluster it on your own create a pie chart create insights. If you've done that once you maybe do it a second time but then it gets difficult. But basically this was the most useful thing I did in the end because it was a huge time investment but in the end I ended up with a pie chart that really illustrated customer problems and it really illustrated problems in our self-service architecture that we had for this product. So a lot of customers like two digit percent canceled their account because they they were stuck with a problem that could not be solved on its own and customer service couldn't solve it for them. So they basically had to cancel their account and hopefully create a new one. So that's the data side of things. So that's if you have this data you can now start planning your initiatives and the first thing is just in hindsight going three months up up front from the point where you are right now with your data don't focus on the first thing. Invest the time take your team small meeting get around get a brainstorming whatsoever ideally get a whiteboard best tool in the world and write down everything you could do and discuss it. Build a backlog of these things that could solve it and then go through a process of selecting the thing you want to start with. Last week we talked to Josh about problem statements and hypothesis and I know that you are also a big fan of it. Could you elaborate a little bit here? Sure sure yeah that's basically what you create on the whiteboard is ideally not solutions directly what could we do you already because you have the data you're building a hypothesis you have you say 20 percent of customers have problem x that's why they cancel the account and now you're saying if we build functionality y then they will not cancel their account. So that's forming the hypothesis basically out of this two activities of creating the data set and thinking about solutions these both put together is your hypothesis that you're going to test and automatically then the next step is defining the success criteria. If we build this and this is going to be successful we will see after implementation waiting a couple week the number of the percentage of people canceling because of this reason because we have the data now and we're collecting it on the go will go down if not it failed I didn't understood the system next iteration. But this is very important if you just start with step two and say we're building this I know already why they're churning you will not be able to measure you will look at the overall churn number you will see a little tiny dip of it but in this time a market change occurred a competitor came in people churning for completely different reasons and the impact of your feature is basically gone. Doing this I would call it product due diligence before you actually have your bias of action everybody is talking around and running and going and build things right away it's very important to to do this and do this as effectively as possible and involving the right people that can help you because it's not always that trivial as that. So understanding a little bit of statistics a little bit of probability gets you a long way because usually it's it's not like simple as the most simple fact that we are discussing here is customer churning it's it's a one or zero but it gets more fuzzy look at customers for example if you look at screen time for example you want to increase the amount of time they spend on a certain screen but what is your hypothesis for example is I change the screen background color to red then they will get confused and they spend more time on the screen I reach my metric but here you cannot look at percentages a very important concept to understand the problem you're dealing with you're dealing with probabilities and thresholds here and that's where you definitely need to talk to a data person because the worst thing you can do in this sense is set five percent more screen time this is not going to help you here we need to do proper call it there's a hypothesis theory in statistics with p values and all the stuff it's I just recently learned that and I'm a little bit excited about it because it opened another world to me what you want to look at is basically the probability of users spending more than the previous amount of time on the screen and you want to reach a certain probability in b2c and optimizing applications this is bread and butter all the time you will not get away with saying 10% more 5% more you have to dig in probability you have to talk to your data person and you have to design proper analysis and it will get you a long way do this and deal with the complexity that it creates now you have that and you have an idea of the metric you want to move you have to analyze it you have to discuss in a group what are the best things to do here and that forms your hypothesis write it down and discuss now what you want to do and now the question is now we're talking priorities and now we're talking prioritization before we jump into the prioritization of like different problems still on the data because I think that's super interesting especially also on the topic that I'm currently working on what would you say if you look at churn is there a threshold do I need to have a high volume business to be able to analyze churn in a good way or can I also have just a few customers and still being able to analyze it because I think obviously you mentioned 3000 comments coming in on people like leaving a service that that's a difference than if you have a low volume high ticket size kind of business it depends also on your business usually there are many perspectives to look at churn in for example in b2b you not necessarily look at your clients churning you more look at if you say you're a subscription based company you look at your subscription revenue churning away and look deeper from that maybe when you have a you probably have a business with a lot with revenue and you have a limited amount of customers in b2b it's sometimes better not looking at the customers to identify a churn metric it's maybe better to look at other kpis of the business like revenue it comes very much intertwined when a big client churns you lose a big chunk of your revenue at some point then you're on a different realm of research you're not in quantitative research more anymore where what I talked about where you analyze the what basically you have to really dig into qualitative research go into the interviews and understand the details it is very hard you don't need to talk to a data person here you need to talk to a designer or a dedicated researcher that can extract the exact point by someone churned which is also a profession on its own don't do it on your own just to make a sanity check on where we stand we have an issue we realize that customers leaving our company for whatever for a couple of reasons not for whatever reasons so we know 20 percent of our customers are leaving for reason x you sit down with your team you have understood why they are leaving and now you're thinking about solutions one step that i'm missing in between is the point where you define the right kpis you've mentioned for example i don't want to put you on the spot here but you want to increase the screen time is improving the screen time per se the right kpi or would it be for example a better kpi saying okay we want to improve joining the screen until interaction x so i just try to better understand the whole process of finding the right kpis and also asking yourself what do i want to learn from it not putting you on the spot right that opens a little bit of another can of worms it goes back to the question where are you at as a product manager because there is not not talking about science all the time but i think what is very much neglected and all the very high product management books are all about basically the scientific method that you get teached in the school all the time about you do an assumption you do an observation you do an experiment you do your analysis and then you prove your hypothesis wrong and you go all over again steps are interchangeable there is no unification of it but this works on separate levels you as a product manager are might be tasked right now to optimize the churn rate but the business on itself depending on what state it is right now it operates on a completely different hypothesis right now that you need to understand as well in which phase is your business you will not be tasked with optimizing churn rate in a very young startup that exists for two years they are on a growth trajectory if you hopefully otherwise wouldn't made it to two years so what it means is basically you also have to understand the the game you're in one level up so what is the company trying to achieve what hypothesis they trying to prove right now is it product market fit wouldn't look at churn rate probably that much it is probably inserting themselves into a higher level of market share and optimizing the amount of market share that they have so what what is the overarching kpi is probably looking at the market share you have of the certain market the sub kpi is okay we're losing every month let's say healthy churn rate one or two percent of my customers and twenty percent of these customers i lose i lose through these problems and if this is at scale it is worth investing a lot of resources in rescuing these twenty percent of two percent of my customers to improve the amount of market share i have you're saying that defining the right kpi should be always connected to the big picture and also the current strategy or the face the company is in to be able to make the right decisions exactly for churn it's easy to understand it's always good for the business optimizing for screen time on a certain screen is a little bit harder to understand that's important also to to give as a product manager a proper explanation why and a proper motivation and this is always connected to the higher up goals of the company especially i haven't seen many companies that have the perfectly figured out okay our structure yet but that shouldn't stop you and or anyone to say yep just doing that right now because it's my job it's very easy if you apply a little bit of reflection on where is the company at why i'm supposed to do that to make your own assumption on that and say okay 20 more percent time on this screen which is maybe our order detail screen let's say it makes it more likely for customers to put it in a shopping bag and then go from there and from the shopping bag and so on i'm improving an overall conversion funnel maybe so even if it does not give them directly then this is something you can derive from a little bit of deduction using the bigger picture of the company so we will jump to the solutions but before we do that i i have a final question just because you also mentioned okrs and overall company goal where does churn sit especially thinking of bigger product organizations would i task a team to look at churn specifically or would i pretty much should it be a kpi that all the teams look at in their respective like product area oh i i wouldn't make it so i don't want to make churn a thing basically you'd look at churn basically you always as in a startup environment which i'm moving in right now you look at churn when the investor tells you to look into churn because they want to understand their investment better that's the first kpic craft you don't have to expect that you start working in an environment with a perfectly set up data warehouse where every little piece of front-end code you do has google analytics tracking on it churn is something that depends on your organizational structure it can sit with a team that owns products are sliced differently sometimes you have a team tasked for the onboarding experience sometimes you have an engagement team and and then specific teams for specific areas of your product i wouldn't give it to to all teams because an onboarding team cannot really influence churn they have their own funnel and see and have their own dropouts i would give it to the team that has the most control over it when it comes to building the solutions the analysis can be a joint product effort so it's a kpi that you are interested as the product group as the company and then depending on what you learn from your data you can also bring the individual initiatives to the specific teams when you have the motivation what you want to optimize for and then they have a very clear metric which is also nice you can perform against with a very straightforward problem to solve and basically with underlying metric you want to improve it sounds very straightforward but not all teams have that all the time so this is something i always look out for and i'm very happy about when i have it and can give it to a team talking about teams once you have realized what you want to do you said it's time to prioritize things which very likely will happen also within the related team in case you are working cross-functional or even with a separate one but the point that i would like to discuss before is how do you make sure that the team has something to prioritize hashtag feature preparation let's say it this way it doesn't the team doesn't need to always have something to prioritize they need to have a problem to solve and the overarching problem is we have customer churn and here are the reasons why or here you can even drill down if you want to give it to multiple teams then here's the problem why 20 of the customers that leave our product leave it and that falls in your area would you agree yes okay please work on solving this problem so that's the first thing teams should always have a joint discovery i wouldn't go to an engineering team saying here's a solution we have five solutions please implement them in this order but you don't create ownership in a team in this way so basically if you give them a very clear problems with boundaries with a metric then that's what they need to do okay but from my experience you're working in a team and you're usually not only fixing one problem so you have multiple problems with different people different stakeholders who have different requests so how do you evaluate what comes first and how do you prepare your team to make the right decisions on building the right things first why i said building up a backlog on this specific topic is the is it's called an ideation step how you could solve this problem you will create things that are hopefully outrageous challenging innovative that you will never be able to implement you might not want to file them in a gyro epic and so on what you want to keep them for later or to have just a record of the due diligence you did because in five months when you improved on the metrics you moved on there is a messaging slack someone tells you why didn't you do that if you can trace back and said we decided to not do that for a reason a b c that gives you a lot of cred as a product manager it proves your your professionality do that write it down have it so now prioritization in team how i prioritize my experiment or implementation to reduce the churn over other things that are coming you have a roadmap you have priorities so when you have the metric underlying you could very quickly estimate the user value to gain from it well i'd rather say the company value also use a value in the sense and if you have that in an ideal scenario which not exists you have it for all your other features then it would be very easy data driven prioritization what i think is very helpful there is this very old project management triangle that i see quoted all the time like time money resources and so on how you can balance things it doesn't work for product management very well because these are projects when they end everybody moves away project is gone for product what works for me quite well as a mind model is to think in in constraints that i have so there's first the it comes from production it's nothing fancy i try i adopted it a little bit you have a certain user value that you can deliver but you also have resources that are required for that for a user value you gain something with your users you could call it money or so on the resources cost you money and everything you do gives you an operational overhead or maintenance also cost you money in the end you should always prioritize the user value but you need to make compromises and one of these constraints will always limit your overall system like the weakest chain of all of it what happens very often when you're an unexperienced product manager that has that is a very good communicator he proposes a solution that has that looks like the best one but doesn't address all the edge cases that come with it it is the super it's the easiest to implement but he completely ignores the operational cost especially as a product at scale that comes with it so you implement it you cannot take it back it breaks all the time it causes a huge customer service overhead then you gain basically nothing in the end because your team is dragging this along needs to either verbally help support out fix it rework it and so on so when you think in these three dimensions and take your backlog this way what gains me the highest user value what is the most costly and where i either create operational costs and where do i reduce operational costs it is one perspective of many to look at your backlog but then you can assess where are you in with your team there are teams that have not much software built yet so they have low maintenance that's very easy for them to say we build something that costs us less resources and creates a huge user value and we take on more maintenance for it if needed we don't know yet but if you have a team that already managed multiple services has a lot of dependencies already you need to be very careful when building something to not increase your operational cost otherwise your team will drown you need more people to maintain it or you will be stuck in big big refactoring at some point so you're looking at value cost in that case and i assume you are drawing it on a matrix to realize okay these are the the cash cows so to say and then you go for it right yeah usually it's easier you don't need to draw it or make a data driven assessment of your priorities all the time most of the time it will be very clear to you what is the most important thing to work on because it's either given by general priorities of the company or you have a good gut feeling you will develop it and also it depends sometimes on your team when your team went through a slump refactoring a service for a long time working on a very important project for a company but that was dragging on and was not fun for the team it is sometimes also as the pm very important to prioritize in a way that the team can work on something engaging over another boring but important thing so if you have the wiggle room for it and i think team and also keeping the team happy brings me a little bit to the final topic that i would love to discuss in this round because we've talked about your background you are a product designer or you've been a product designer before you became a product manager so with the product designers glasses on like how do you actually empower designers or how do you involve them in this process of discovery and ideation and down to the delivery discovery ideation ideally they run it with me or for me ideally no it's very difficult and depends to person to person how to work with designers because they are in one area they are the biggest asset you have to create innovation and drive creativity and even motivation in your team and on the other side there can be a huge pain in the ass when they go completely off course for a week or so and don't go back basically you kill creativity in your designer if you as a pm be too restrictive with what you expect from them if you go to your designer with a detailed wireframe that you built after work and never tested in a prototype you just draw it you said to the designer this is what i want to have please color it for me this designer is not going to last in your company at all but on the other side if you're to wake with a designer and say we have churning customers do something then you will not have a tangible result expect you have one of these unicorn designers that can manage him or herself very well has business in mind and so on then they deliver it to you in a week but the normal case is if you are too awake they will go left and right and the result will probably disappoint you and take too long so it's important to find a proper partnership agreement how you want to work and actively give each other feedback when someone is overstepping too much or especially a designer you can tell a designer look i need you to work on this thing right now and there's a team waiting to implement it next sprint so you have some responsibility here and we need to drive the two conclusion what do you need to do this for example so it's important also to hold designers accountable even though they're creative unicorns all the time and when they open their mouth butterflies fly out everything but this is this is the most important thing working with designers keeping the creativity up but making your constraints clear that you have for this design we need to solve this problem this problem only let's take the other one to rethink the navigation structure of our application maybe but i think obviously like we're with design background and you're not the only product manager who has the background or maybe there are product managers who don't have design background but think they're good designers so yeah yeah i remember that it happens pretty often i would say that pm has an idea and he starts already like designing it and wireframing or that it's like pretty clear and i think if you even have the skills to design it it's very easy to be like okay i have all the information that's what we need to build i just design it and then pass it on how do you deal with it is there also a moment where you just say okay no i'm a pm so i just need to actually give them the freedom how does that work oh i wouldn't so i think what also separates a regular designer from a senior designer the main trait i would look for when hiring a senior designer is that they stop being attached to their work at all and always have a good portion of self-doubt i think i got to this point as a designer and self-doubting everything i do and are very open to look at like critique and feedback and questioning myself and throw everything away and start over so that's that also makes it very easy for me to not be biased on my own idea in general if you go very pragmatic about it you can direct design when you for example work on with designers actively on an information hierarchy or a very rough flow say process flow that you want to achieve that you can agree upon first so that limits the risk that the result that you would expect even though you don't really know what to expect goes into the wrong direction or will disappoint you of what you expect basically so what i used to do is basically work on information architecture a lot that helps me to say okay here are the things this is how i would structure it and this is the result i want to see on the next screen so let's take a very a problem that i had recently we wanted to switch the state of something basically it was green before a was green and now i wanted to switch it to b to be a screen and there were dependencies a could be switched to b and c and then d could be switched to a and b and it's a very interesting ui problem to solve how to bring give that as a very easy understandable form to the user so what the designer initially did was basically building huge forms you have to have check boxes what you want to switch to what and so on which i was not happy with and then instead of redesigning everything i told him look can we simplify it what's the actual core here that we want to achieve users should switch from a to b they don't need to know about the other things from a process and architectural point of view we want to achieve a to b and c to d basically can we abstract it you didn't know that you could do it because i haven't told you before that this is a possibility that we not need to show all the information that is available to to the user so let's open that up and when you talk in this higher level you still can make an agreement how the process should function without impacting the ui too much i know it was very abstract i mean the a to b to c to d because maybe just while it was complicated i i think like the messages that you don't need to talk about ui right it's not that you need to tell him okay design it like this it's more about keeping it to the abstract level or for example to the level of information architecture as the most detailed one to get everyone on the same page while still giving them the freedom what is also very powerful in this discussion with your designer is as same as you do in engineering writing pseudocode writing the interface is extremely helpful for a designer to understand what should actually happen and then a lot of mistakes will never occur basically if you write down what the user should do and you agree you know in written form on what you i should do not bullet point acceptance criteria it can be narrative sentences shouldn't be a page shouldn't be few then you can pinpoint it down and often depending on your the more you abstract you make your design draw it on a piece of paper instead of using a tool and abstract it as much as you can from the current ui then you're also not impacting the creativity of your designer which also means like generally having an iterative approach or process for designing things which i think it comes down to like really forcing people to not jump to the final ui immediately because i think that blows everything up and that makes things start makes things super complicated but when you keep it on this high level paper sketches and so on you can get an initial agreement and then drill down into all the different levels when your ui doesn't explain it in some point is it's to to the fact of iterating on design it's also very important to understand that depending on your definition of ready in the team differs from company to company i favor teams that are or try to build teams in a way that are a little bit more flexible if you have an agreed up on flow development can already start and you gain speed and the final ui can happen in between because when you have this agreement on how it should work and you discuss this extensively the the visual design part of it what polishing or coloring of it is 10% of the work and also 10% of the work for the engineer so this is usually not where the complexity sits basically the user experience part is is the critical one where you should align with your designer fully agree on that i would say enough a to b let's talk about zero to one matzen i really like the example and i would like to hear to finish this nice round today what are the two biggest learnings you've made in that particular churn project first the freeform comments of course only also when you work with a data team product ties your data needs do the same thing you do the same scrutiny you do for when building something with engineers when requesting something from your data team don't say oh this metric would be nice when i get my morning coffee i want to see a dashboard full of metrics then mean nothing to me what i used to do is also just prototype the metric if i can i get the raw data if it costs nothing i built an excel dashboard and battle test myself if i even look at it for a couple days and if it turns out useful then we productize this metric even so before that data people spending time and are disappointed when the data is not used this point one and point two actually is underestimating the impact you can make through that because in the end we ended up by overall reducing the churn of the product by 40 percent when we only estimated to make a 20 percent increase on this specific project we implement we chose the functionality that had a lot of synergies with other problems it was a little bit cumbersome to implement was a journey but in the end it turned out very nicely and we could only know that it turned out so nicely because we had the initial data we made a success estimation and then we exceeded it and for a team to put on a yearly report or for the next okr store to reflect over the year this is uh super important when the product manager worked in this framework and can bring this achievements or or illustrate the achievements with that better amazing what a beautiful closing word for the session i would say there is not much left than thanking you and i think we can finish our glass of wine after the debrief exactly thanks matzo thank you thanks for having me so all right so i suggest we keep it short this time we have anyway it's an in-person meeting so we have our guest sitting here we have the glasses full uh with water i think actually one thing that i always wanted to bring up in a debrief i mean for everyone who listened this far right this means you really enjoyed the episode i think it's always funny that we debrief at the end and we did try to have this in the middle of the session or at the beginning of the session but i'm like maybe we can use this place to generally make a shout out to get some feedback are the debriefs actually helpful for you especially as you're still listening should we keep doing them should we give our summary should we change the format or should we move them again back to the beginning let us know with that said i think the the whole episode was actionable i think it's definitely an episode also after we have added the table of content where you can go regularly through and just relisten again because there were a couple of things through a couple of topics and actually matthias explained a big part of the product discovery and development process so i can highly recommend listen twice if needed and if you have any questions you can also just reach out to matthias via our website product-bakery.com slash episodes and one thing if you're a pm and you work with designers bookmark this episode as well i think there are some good tips for collaboration and lastly in case you have a cool story to share and a case study that you want to discuss with us in person or remote let us know drop us an email at hello at product-bakery.com and we will get back to you and maybe end up here in one of the live episodes awesome thanks everyone and have a beautiful day and night bye