24 Sep Improving steering or adding a new feature?
“Is our steering that bad?”
Harry is Yimian’s client. He is an engineering leader in the R&D department of a leading luxury car brand H.
He had been struggling in prioritizing R&D spending for 2019-2020. His team was truly excited about pushing a new feature into production. However, steering problems had come up several times in recent cross-functional business review meetings. Customer support leader had been anecdotally raising issues on steering.
Harry agreed that steering needs improvement. However, should H prioritize steering over a new feature? Specifically,
- How well H was doing in steering against the competition?
- What specific steering issues were customers talking about?
- For each specific issue, were customers talking positively or negatively?
Of course, Harry could look into service data aggregated from Brand H dealership stores. However, that dataset wouldn’t provide much insight into the competition. Additionally, he could ask the Market Intelligent department to design and conduct a steering-specific consumer survey. That wouldn’t happen in time for annual budgeting, given the long backlog of other higher priority surveys.
That’s when he approached Yimian. Here was how we provided an answer to Harry.
1. Collecting comments on competing products
First of all, we agreed to collect customer posts from China’s No.1 auto discussion forum.
The auto forum is organized by brand, series, and models. For example, Buick-Encore-2020 CVT Luxury. We aligned with Harry a few direct competing brands to H, and collected customers comments and replies under each brand’s forum.
2. Removing fake comments
Not all comments are equally valuable. Chaffs need to be removed from the wheat.
Before we conducted any analysis, the raw data needed to be cleaned of any astroturfers. Astroturfers are paid, fake customers. They repeatedly rave about the sponsors’ brand and products since most of them are paid by the number of fake positive reviews.
Spotting one or two astroturfers is easy. If a specific user ID posts a large number of highly similar comments with strong positive emotions, it’s a highly suspicious astroturfer account.
However, doing this manually across hundreds of thousands of comments would be unrealistic. Therefore, we automated this task using an “astroturfer removal” algorithm. It identifies suspected astroturfer accounts by frequency, similarity, and sentiment. We also added other dimensions for other clients in the past, such as frequent posting time in the day, posting intervals, or the age of accounts. For Harry’s project, we used the 3 most telling dimensions: frequency, similarity, and sentiment.
3. Extracting actionable topics
Customer posts are loose texts. How can we extract meaningful numbers from them?
After removing suspected astroturfers, we turned unstructured natural comments into structured quantitative data using Natural Language Processing and an automotive knowledge chart. The knowledge chart includes 11 level-1 topics such as handling, ride, fuel economy, price, etc. These, in turn, include 45 subtopics and 245 level-3 topics. At level 3, we are already talking about very actionable granularity such as “dealership follow-up call?” or “noise from windows”.
For each level-3 topic, there are multiple phrases talking about the same thing. For example, “rattling”, “noise”, and “strange sound” are equivalent in the context of “noise from windows”. With the help of a “synonym finder” tool, we cover the most common phrases of a certain topic.
With this knowledge chart, we parsed the forum posts, extracted steering related mentions for all target brands. There are 6 sub-topics including low-speed steering force, high-speed steering force, steering noise, steering vibration, precision, and feel.
4. Sentiment analysis
The next step was to see whether or not customers were happy about a level-3 topic.
We used sentiment analysis to count mentions with positive, neutral, and negative emotions. Borrowing the concept of Net Promotion Score, we can compare the percentage of net positive mentions among all mentions of a certain topic for all brands.
At the end of the project, we showed Harry that brand H actually had the best feedback on steering overall in the last two years, compared to its major competitors. We also identified “steering precision” as H’s most competitive attribute, and “steering noise” as its least competitive attribute.
Taking these results to the cross-functional meeting, Harry was able to make a stronger case to prioritize the new feature before improving steering, and to spend some laser-focused budget for steering to address the noise.
Please email me (firstname.lastname@example.org) with your challenges to see how Yimian can use auto forum data to support your decisions faster with higher resolution.
(Cover photo credit: Landon Martin)