Microbrowsers are Everywhere

Colin Bendell gets into the minutia of microbrowsers - the small previews of your site that are pervasive all around the web and through social media apps and search engines whenever an item of content on your site is referenced.

You’ve seen it everywhere - that little thumbnail preview of a website mentioned in a tweet, the expanded description in a Slack channel, or in WhatsApp group chat.

Figure 1: The preview shown in a group chat provides a hint of what the real webpage looks like

These link previews are so commonplace that we hardly pay any attention to how our site design might be impacting the generated preview. Yet, these previews can be the most influential part for attracting new audiences and increasing engagement - possibly more than SEO. Even more alarming is that most web analytics are blind to this traffic and can’t show you how these Microbrowsers are interacting with your site.

As we close out the year, here are five essential questions and ideas that every web dev should know about Microbrowsers.

1. What are Microbrowsers? How are they different from “normal” browser?

We are all very familiar with the main browsers like Firefox, Safari, Chrome, Edge and Internet Explorer. Not to mention the many new browsers that use Chromium as the rendering engine but offer unique user experiences like Samsung Internet or Brave.

In contrast, Microbrowsers are a class of User-Agents that also visit website links, parse HTML and generate a user experience. But unlike those traditional browsers, the HTML parsing is limited and the rendering engine is singularly focused. The experience is not intended to be interactive. Rather the experience is intended to be representational - to give the user a hint of what exists on the other side of the URL.

Creating link previews is not new. Facebook and Twitter have been adding these link previews in posts for nearly a decade. That used to be the primary use case. Marketing teams created backlog items to adopt different microdata - from Twitter Cards and Open Graph annotations for Facebook. LinkedIn likewise embraced both Open Graph and OEmbed tags to help generate the previews

<meta name="description" content="seo description long">
<meta name="keywords" content="seo keyword list">

<link rel="shortcut icon" href="favicon.ico" 
                          type="image/x-icon">
<link rel="icon" href="favicon_32.png" sizes="32x32">
<link rel="icon" href="favicon_48.png" sizes="48x48">
<link rel="icon" href="favicon_96.png" sizes="96x96">
<link rel="icon" href="favicon_144.png" sizes="144x144">

<meta property="og:title" content="Short title here" />
<meta property="og:description" content="shortish description" />
<meta name="twitter:title" content="Short title here">
<meta name="twitter:description" content="shortish description">

<meta property="og:image"
      content="https://res.cloudinary.com/.../hero-img.png" />

<meta name="twitter:image:src"
      content="https://res.cloudinary.com/.../hero-img.png">

As group chats and other collaboration tools have become more prevalent, we have seen many features from the big social media platforms emerge. Particularly in recent years we’ve seen the adoption of the link unfurling behaviour in these chat platforms. Rather than reinventing the wheel, each platform looks for pre-existing microdata to generate the preview.

But which data should be used? How should this be arranged? As it turns out, each platform behaves slightly differently; presenting information in slightly different ways.

Figure 2: The same amazon link shared in iMessage (left), Hangouts and WhatsApp (right)

2. If Microbrowsers are everywhere, why don’t I see them in my analytics reports?

It’s easy to miss the traffic from Microbrowsers. This is for a number of reasons:

First, page requests from Microbrowsers don’t run JavaScript and they don’t accept cookies. The Google Analytics <script> block won’t be run or executed. And all cookie will be ignored by the rendering agent.

Second, if you were to do a log analysis based on HTTP logs from your CDN or web stack, you would see a relatively small volume of traffic. That is assuming you can identify the User-Agent strings. Some of these Microbrowsers impersonate real browsers and others impersonate Facebook or twitter. For example, iMessage uses the same User-Agent string for all these requests and it hasn’t changed since iOS 9.

User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) 
             AppleWebKit/601.2.4 (KHTML, like Gecko) 
             Version/9.0.1 Safari/601.2.4 
             facebookexternalhit/1.1  
             Facebot Twitterbot/1.0

Finally, many platforms - particularly Facebook Messenger and Hangouts use centralized services to request the preview layout. This, in contrast to WhatsApp and iMessage where you will see one request per user. In the centralized consumer approach your web servers will only see one request, but this one request might represent thousands of eyeballs.

3. Microbrowser are probably more important than google bot

We all know the importance of having our web sites crawled by search engines like googlebot. These bots are the lifeblood for lead generation and for discovering new users.

However, the real gold for marketers is from word-of-mouth discussions. Those conversations with your friends when you recommend a TV show, a brand of clothing, or share a news report. This is the most valuable kind of marketing.

Last year when assembling the data for Cloudinary’s State of the Visual Media report, I discovered that there was a very prominent usage pattern over the USA holiday season. During thanksgiving, all the way to Black Friday, the rate of link sharing skyrocketed as group chats shared deals and insights.

Zooming out (and normalizing for time-of-day), we can see that there is a daily cadence of link sharing and word of mouth referrals. It probably isn’t a shock to see that we predominantly share links in Slack between Monday and Friday, while WhatsApp is used all week long. Likewise, WhatsApp is most often used during our ‘break’ times like lunch or in the evening after we put the kids to bed.

While the link preview is increasingly common, there are two user behaviours to balance:

Users can be skeptical of links sent via SMS and other chats. We don’t want to be fooled into clicking a phishing links and so we look for other queues to offer validation. This is why most platforms use the preview while also emphasize the website url host name.

Skimming. I’m sure you’ve had the experience coming out of a meeting or grocery store to find a group chat with 100 messages. As you scroll to catch up on the conversation, links can easily be skipped. In this way, users expect the preview to act as a summary to tell them how important it is to visit the link.

Figure 4: Nielsen Norman Group summarizes the research in a dynamic image preview

Figure 5: A mockup of how an ecommerce product could create compelling previews showcasing colors, stock and price in the preview

4. Microbrowsers are not real browsers (they just play one on TV)

As I previously mentioned, Microbrowsers pretend to be a browser in that they send the right HTTP headers and often send impersonating User-Agent strings. Yet, there are several characteristics that a web dev should be aware of.

First, Microbrowsers try to protect the User’s privacy. The user hasn’t decided to visit your site yet, and more importantly, the user is having a private conversation. The fact that your brand or website is mentioned should just make your ears burn, but you shouldn’t be able to listen in to the conversation.

For this reason, all Microbrowsers:

don’t execute JavaScript - so your react application won’t work
ignore all cookies - so your A/B or red/green cookies will be ignored
some will follow redirects, but will quickly time out after a few seconds and give up trying to expand the link.
there won’t be a referer: HTTP header when the user clicks the link for the full browser. In fact, a new user will appear as ‘direct’ traffic - as though they typed in the url.

Second, Microbrowsers have a very small brain and very likely don’t use an advanced network algorithm. Most browsers will use a tokenizer to parse the HTML markup and send requests to the network stack asynchronously. Better yet, browsers will do some analysis of the resources needed before sending the async request to the network.

Based on observational experimentation, most platforms simply use a glorified for loop when parsing the HTML and often request the resources synchronously. This might be ok for fast wifi experiences, but it can cause inconsistent experiences on flaky wifi.

For example, iMessage will discover and load all <link rel="icon" > favicon, all <meta property="og:image" images, and all <meta name="twitter:image:src" before deciding what to render. Many sites still advertise 5 or more favicon sizes. This means that iMessage will download all favicons regardless of size and then not use them if it decides to instead render the image.

For this reason the meta markup that is included is important. The lighter the content, the more likely it will be to be rendered.

5. Markup Matters

Since Microbrowsers are simple-brained browsers, it is all the more important to produce good markup. Here are a few good strategies:

It’s almost 2020, you only need one favicon size. Remove all the other <link rel="shortcut icon" and <link rel="icon" references.
Based on observational experimentation, the most commonly recognized microdata tags for preview are the Open-Graph tags. When the OG and twitter card tags are missing, the default SEO <meta name="description" is used. However, since the description is often nonsensical SEO optimized phrases, users’ eyes will likely glaze over.

On that note, use good descriptive text
Provide up to three <meta property="og:image" images. Most platforms will only load the first one, while others (notably iMessage) attempts to create a collage.

Figure 6: Amazon uses User-Agent detection which results in many link previews using the description meta tag.

Use <meta property="og:video* with progressive (not streaming) video experiences.

<meta property="og:type" content="video.other">
<meta property="og:video:url" 
               content="https://shoesbycolin.com/blue.mp4">
<meta property="og:video:secure_url" 
               content="https://shoesbycolin.com/blue.mp4">
<meta property="og:video:type" content="video/mp4">
<meta property="og:video:width" content="1280">
<meta property="og:video:height" content="720">

Don’t use UA sniffing to hide the <meta> tags. Sites like Amazon do this to try and show only Facebook/Twitter the microdata annotated website. But this can cause problems for some Microbrowsers that don’t use the same impersonation convention. The result is a simple link without a preview.
Use the opportunity to tell your product story or summarize your ideas.

Summary

As more of our conversations happen in group chats and slack channels, link previews are an important way for you to engage users before they start the journey on your site. Unfortunately, not all websites present good or compelling previews. (And now that you know what to look for, you won’t be able to unsee bad examples - I’m sorry). To help users take the leap and visit your site, we need to make sure that all our pages are annotated with microdata. Better yet, we can use these previews to create compelling visual summaries.

About the author

Colin is part of the CTO Office at Cloudinary and co-author of the O’Reilly book High Performance Images. He spends much of his time at the intersection of high volume data, media, browsers and standards. He recently helped the community effort writing chapters in the Web Almanac on Media and CDNs. You can find him on tweeting @colinbendell and at blogging at https://bendell.ca