# Bolt ## 10/03/2022 ### First Meeting Takeover from Travis * Engineering on Bolt side will have more time to in app experience * Goal for in app by 31 October * Vera to talk about another issue: * Retraining & no access to courses * Google Sheets collecting information * How many cases and tickets are coming in per week? - 300 * Able to complete it, but "blocked" * Open the course but redirected to Bolt home page * Or can't open the course * South Africa & Tunisia - 7 markets total * Could be network connectivity * These markets tend to have older phones with much older versions of Android * iOS defaults to Safari, Android uses "web view" which is old and could not work well with react framework that Northpass uses * It is curious that this is happening with retraining. * Charlie - the only thing I can think of doing is reverting the look to an older look * Single example of Azerbaijani with OS 11, but not clear what the actual issue is * Shows as complete, but he claims he could not complete. This sounds like a different issue. * Nigerian driver running 8.1, this is the common issue and it _seems_ to be an OS versioning issue * Possibility of the UID/Email not getting added in correctly? ## 11/14/2022 ### Bolt In-App Training Release * Launching in four markets: Azerbaijan (two languages, RU, AZ), Nigeria, Kenya, UK * No training exists in France * France and UK you can't mention "training" they are all suggestions * "All is under control" * Vera is back on Wednesday to talk about Arabic course. * Everyone is waiting anxiously for in-app training * Contract ends in January, she wanted to release earlier to have more analytics before renewal * If it goes well, more markets will want this training. * When everything is in-app, she's expecting higher adoption * Do African markets adopt differently during the Winter Holidays? They may not celebrate Christmas. ## 11/28/2022 ### Post-Android Release Meeting Release of the new courses keeps being delayed. Bolt engineers sent video of what the in-app training looks like, but the drivers see the category of the course. However, the course category is just for Doris. The drivers should not see the category on the course overview page. DONE: Hide category from course overview page. Doris also getting asked about the in-app learning launch. She asked about Crowdin - a translation service. Do we have an integration? DONE: Research a Crowdin integration. Ask Vera what her workflow is like. ## 12/8/2022 ### Prep for QBR The issue from Joonas, all in African Market. Mostly in Nigeria. Azerbaijan, SA, Tunisia, etc. The recording is from someone who actually came to the hub and recorded their screen. What Bolt can see is the attempts which will show/not show if people are completing or having trouble. From launch, 5 people have not completed it. They were validating training vs no training in each markets. Acceptance rates and reviews are lower for those who haven't completed training. This pilot is the first time we are able to test training within the app. Non-Trainees have to figure out the system on their own. Trainees are told to accept everything, and so their acceptance rates are higher. Non-Trainees ask other users, i.e veterans, and have more street smarts and therefore earn more than people who take the training. To better grasp the content: * They took this content themselves, and have A/B/C/D course tests. * Testing Internal Platform (called Stories) and how it relates to NP and courses. * Stories is a set of slides, and they are beginning to leverage it for learning. * Group D is Edume. They don't have a contract, but wanted to see the test. * The three platforms look very similar. Designer created a similar experience so people all have the same experience. * Stories is marked as completed by going to the last slide. * Average Training Attempts, Northpass is the worst. * Started attempts... something is causing the users to restart. * Northpass, median minutes of completion is the worst. * If a driver doesn't organically reach the last step in registration, they don't get enrolled in the courses and are locked out. * Usually they don't reach if they don't offer some documents. * Training is sitting on the last step. So without the steps. * In-app training should solve this last issue due to them already being registered and in the app. * Median minutes is very preliminary since its only been 8 days. This first is from enrollment to completion. * Median minutes for spent on training is the entire course. Not activity. * Out of 10 cases of average start count is > 5... more than 5 times they have tried to start training. 10 cases total, 8 are Northpass. DONE: Look in DataDog after Doris sends time or people that are being stuck. Something else that Joonas has mentioned, enrollment process between Northpass & Edume is quite different. Differences: * Northpass needs first, last, and email * Edume doesn't need any information. This is helpful with GDPR. By next Friday, Doris is hoping to have more data and details to see how the previous data changes. What Doris is happy to see, is that the activation rate of non training users is much lower. So training has specific and real effects. Between Markets, Training vs No Training is quite different in Azerbaijan and UK ## 12/15/2022 ### Bolt Prep We have made 3 changes since their renewal: * Updates * Better CS * Improve Platform Capabilities We have made updates to the course player. Redesign Course. Help build Rich Text content. Setup in-app learning. We've accomplished everything that's been asked. DONE: Build Slide deck for Sten Data Recap, Reports to Pull: * Data before Course Player Upgrades * Improvements after Course Player Upgrades * Improvements after end of course screen ## 12/16/2022 ### QBR Sten: Technical Reliability is an issue. What are the specifics? * Re-training is causing issues * Blank Screen in web view is specific to Android Web view Issue * Blank Screen for "many months" might be an internet connectivity issue. * Time in Training - is this good or bad? Northpass loads times are longer than all other trainings. Why? Baseline? They have discussed - UX researcher thinks the training should take under 10 minutes, but in reality it is closer to 5 min. Kenya and Nigeria have the highest training attempts * Northpass time is 10, others are 4 and 6. Sten's biggest worry is reliability. re-training really impacts the business. AB Testing results show abnormality with Northpass platform. ## 01/11/2023 ### Mobile & App Credentials * Just maybe ask them to thoroughly check how their app is working on mobile browsers. * Check browser logs from mobile device if possible. * We clearly see the difference between mobile and desktop browser behaviour, and the link we’re using is not a reason as we checked with you. Did Joonas fix according to that SO article that Charlie sent? Supposedly, yes. It is difficult to see the impact right away. For them to see an error with a driver is two ways: 1. Start rate > 5 2. Driver contacts customer support with error. If a driver has registered, then they have completed the training and all is working successfully. Joonas will be back on Monday to give us the credentials. ## 01/23/2023 ### Next 6 months Discussion Sten & Oliver have been added to this meeting * Olivier moved to the Training team and took over for Vera and all her tasks. * Vera moved to a different team. Notes regarding contract: Travis wants to review expectations for the next 6 months. Last review was July, Travis thinks it was a successful process. Northpass is aware of some outstanding issues on reliability that need to be completed asap. Do we need a new scope of work? Course related issues have been solved, but platform reliability still needs to be solved. Charlie's webview article suggestion didn't work because of Android issues and versioning. Lots of Drivers are using Android < v10. Drivers are still getting "stuck" - they usually see the white screen. The issue doesn't seem to happen in UK or European markets, only African markets. Internet connectivity could be an issue, but how can we possibly diagnose that? But, for some reason, this is _only_ happening in Northpass. So even if it only happens in African+Azerbaijani markets, it only happens in Northpass. Happens in primarily 4 courses. They are all under the category 'in-app driver training'. They are also mandatory courses, so the issue is more likely to appear. Edume is also being displayed within a webview of the Bolt Driver app. This issue becomes more prominent with re-training courses. Let Aleksandr know about credentials. Travis: other than the webview issue, above, are there other concerns about Northpass? Doris/Sten: Completion times & re-taking count. In Nigeria and Kenya, the minutes spent on NP training is close to double for internal solution. Example: Northpass-11min, Edume-6min, Stories-4min. Across all markets. Enrollment to Completion would be "longer". We are unsure if this is enrollment to completion webhook - or something else? Median minutes is only for people who have completed the training. Moment 1: clicking button in their app, End Moment: Webhook reaching. _The main issue seems to be around webhooks. If webhooks are taking 3-5 minutes to send, then we should be in line with the other platforms._ Next steps - Travis, do you want to create another SOW? How do we want to work forward for the next 6 months? Sten: we need to properly scope the issue that we are discussing. They are being impacted on many fronts from this delayed webhook issue. Once we have alignment on solving the webhooks, then we can properly create a scope for the next 6 months. ## 03/06/2023 ### Doris/Olivier Question about authentication screen: asked if email could be optional In african markets, emails are not used normally. So can they not use one? ## 07/05/2023 ### Bolt Churning & Retention Attempt Sten: they have reached a stage in their lifecycle where they have tried various platforms, including their own. Looking at how information and content is displayed. Saw best results in their own platform. No other use cases for Northpass at all. Need to off-board within one month. 7/31 is the end of contract. Travis: is this economic, just the stories platform. Sten: Bolt is a "super app" and has 5 verticals in a single platform. Because of this they need to display triggers from native in-app information It makes it so much more flexible when using everything within the app. Bite-sized and mini information display is easier within their own app. Doris: Drivers want really tiny bite-sized pieces of information, not a proper course. Travis: Does the name "stories" mean something like Doris: We look at driver behavior before and after they watch those stories and see an improvement in training. Sten: We can't really "train" drivers anymore in the sense of public policy etc. We can't "demand" people to do training, so the training metrics don't really matter. #### Offboarding Specifics Doris: No support needed at this point. They will be making a big global change next week, so we'll see a huge drop in learners in Northpass. DONE: Norm to hold a weekly meeting on Monday in case Bolt needs any support. Aleksandr is the global procurement manager. Wants to sign a termination form, but email is good enough. He is the POC for many of the verticals.