Super Fast Mobile Redirects

Over the past few years I’ve been heavily involved in mobile.  It started with experiments in HTML5 video 4 years ago, and it’s grown to be my bread and butter.

One of the things I realized early on with any kind of mobile advertising is that fast mobile redirects matter.  And DNS latency really matters.  Both of those are things that on desktop aren’t a huge deal… it’s more important that the redirect just work than how fast it works.

But mobile internet time isn’t the same as desktop internet time.  If a desktop redirect takes 600ms on a fast, dedicated connection, that same redirect might double or triple (or more) over a slower, far less robust mobile connection.

And you don’t have to play the mobile game very long to realize that slow link resolution = lost eyeballs = lost revenue = inefficient ad spends.  If you’re running mobile ads, just look at your discrepancy between paid clicks and what registers on your analytics.  The breakage can be enormous.

2 years ago I started working on a fast new mobile redirect engine.  And it worked really well.  Several months ago I had the opportunity to rewrite it.  Version 2 is even better.  And it’s fast.  Really fast.

I won’t go into the technical details of how my mobile traffic director is implemented.  But I will share some of the stuff you need to be watching and looking at for optimum speed.  And, of course, everything that applies in mobile also applies to regular desktop redirects… just without the urgency.

First, let me give you an idea of the scale I’m working with.  Because this level of optimization might not be necessary if you’re running 100 redirects to a mobile offer.  But in my business I ran 1,546,261 mobile clicks through my redirector worldwide last Saturday (6/29).  And at that scale, details matter.

Also, it should be noted that my traffic director does quite a bit more than just redirect incoming links.  It handles geo-splitting automatically, so I can just run my network links to a single endpoint worldwide and handle the destination page dynamically.  It also allows me to do fairly complex weighted split tests.  Because it’s specifically built for mobile, it does device and platform detection.  I don’t currently use that in the redirect logic… but I do pass it on to my destination pages so they can be personalized, if required.  I can also dynamically add tags to the destination URL so I don’t have to modify incoming ad network links (that often require re-approval) just to change a campaign, for example.  Lastly, the traffic director generates an analytics data point containing all kinds of information regarding the click, the source, and the device.

So it’s doing a fair amount of work in the background.

My target is sub-200ms complete redirect negotiation worldwide.  And that includes DNS resolution (which ended up being the final piece to the puzzle).  In the US I actually average < 75ms.  It’s about 150ms in Europe.  And Asia is the slowest, clocking in at just over 200ms.

In my experience, if you want really fast mobile redirects, here are the keys:

  • Proximity.  Get your servers as close to your visitors as possible.  In our case we have 3 server clusters… one in central US (I’m considering moving this to 1 cluster on each coast), one in western Europe, and one in southeast Asia.  Each location has at least 2 servers behind a load balancer for improved robustness and uptime.  After a fair amount of testing, I’ve opted for more smaller servers over fewer bigger servers in each cluster.
  • Geo-DNS.  Proximate servers aren’t of much use if you don’t also have fast, geo-distributed DNS.  As I mentioned, this was the final piece of the puzzle for me.  Until I found a really good geo-DNS provider, I just couldn’t get the entire cold redirect transaction reliably under 200ms.  In my case, I now have geo-DNS with failover to a different datacenter.  That means my links stay active (albeit slower) even if an entire datacenter goes down.
  • A Records.  Ditch the CNames and use only A (IP address) DNS entries.  Frankly, this one surprised me.  Even after adding geo-DNS, we still had inconsistent redirect resolution times.  And most of the lost time was in the DNS lookup.  We switched to A records from CNames and everything fell into place.  I’m very open to CNames not being the actual problem and just a symptom.  What I know is that A records worked well, and CNames didn’t in our setup.  Done deal.  We only use A records now.
  • Measure Locally.  Monitor your redirect times around the world (or wherever you serve links).  You can’t just check them from your office and think they’re working right.  That was probably the biggest testing difference between v1 and v2 of my traffic director.  I assumed v1 was rocking worldwide… and I was wrong.  It kicked a** in the US… and sucked everywhere else.  Local monitoring is important.

The most surprising thing to me?  That it actually matters.  Improving our redirect speed led directly to a noticeable reduction in our ad network breakage.  So more of our paid clicks were seen.  Beyond that, it also improved our conversion rate because we had a lower rate of page abandonment and bouncing.

Was it a huge improvement?  No.  If you’ve got other areas where you can get double-digit movement in your conversion process, you should probably focus on those first.  If you’re down to dotting your I’s and crossing your T’s, you should take a look at your mobile redirect speeds.  They actually matter.

Of course, the biggest downside to all this is the cost.  Just to get a bare-bones service in place with no local redundancy would probably cost > $300/mo.  My setup is quite a bit more than that.  And that’s more than most smaller mobile marketers should be paying for a redirect service.  So it kind of creates a catch-22… you need to be profitably running at volume to do it right… but it’s hard to do it right when you don’t have enough volume/revenue to support the infrastructure.  Unfortunately, I don’t have an answer for that.

Links & Resources (all naked, no affiliate):

After much experimentation, I ended up choosing DNS Made Easy for my geo-DNS.  They’ve been great.  And much more wallet-friendly than other options.  I never have to worry about them and they just work.

I use Pingdom for my ongoing worldwide monitoring and sometimes for spot-checking link resolution time and flow.  I think they’re ok.  There are things I wish they did better… but they work.  And they send notifications straight to my phone when there are problems.

We originally used Amazon AWS for v1 of the traffic director.  After running side-by-side tests, we switched to Microsoft Azure for v2.  We’re pleased enough with Azure that we’ve transitioned everything but some CloudFront content off of AWS and onto Azure.  We have a *lot* of Azure servers now.  Other than a couple of really boneheaded outages on their part, I like them much better than Amazon.

P.S.  I’ve considered open-sourcing my traffic director.  Let me know if you’d be interested in that.  It won’t help at all in the ongoing expense of maintaining a traffic directing service, but it’s a good, bullet-proof, battle-tested solution that works.

C’est la vie

This is one of those posts that I’m pretty sure no one will read. But it’s one I have to write.

It’s been 3 1/2 years since I last posted here. A little longer since I made the most hauntingly bad business decision of my life. A little less since I walked away from the IM community brimming with disgust and anger.

An awful lot has happened in the intervening years. In most ways I’m far better off than I was back then. Business has rarely been better. My 20th wedding anniversary is one week from today. My oldest daughter is starting college in London, and my youngest is entering middle school. From the outside, life is good.

But I have fewer close friends. I don’t trust a lot of people anymore. I stopped helping, sharing, and providing value inside my professional circle (although I share significantly more inside my social circle). And I became even more of a social hermit than I was before, which is saying something.

Mostly, I can’t shake the events of the 6 months following my 40th birthday. I managed to destroy most of what I had spent the previous half-decade building. I ruined my reputation. And I let myself and a lot of other people down.

I think what is most agonizing about how everything went down is how many people think of me and treat me as if I’m a crook. That somehow my evil plan from the beginning was to get a bunch of internet marketers really pissed off at me so they could write and say fantastically crappy stuff about me and never trust me again. (Side note: if you can avoid it in life, don’t piss off internet marketers).

Of course, the truth is far less interesting than the world would have you believe.

In what was probably an act of extreme hubris, I made an astonishingly poor business decision. I made it worse when I chose a business model that accentuated my massive and numerable weaknesses, while not taking nearly enough advantage of my equally massive and numerable strengths. Fittingly, no good “greatest regret of my life” story would be complete without compounding the impact by failing loudly and in public. And when that business started to get off track, I handled it poorly.

By the way, if you don’t have any idea what I’m talking about here, you have a couple of options. One, walk away. I kind of don’t want you to know about this because I feel like I’ve been miscast as the villain in my own life story. That’s my preferred option… you walk away and don’t know this stuff about me.

But the option you’ll probably choose instead is to go ahead and type my name in the ever-present search bar on your browser. You’ll find that a lot of people think I’m Satan’s spawn, come to earth to steal their children and rob them of their life’s savings.

So go ahead and search and find out how bad I am. And if you still want to read more after doing that then come on back… I’ll wait.

(waiting…)

Done? Good. After learning all that wonderful stuff about me, you just might be concerned that even as you’re reading I could be installing a super-secret trojan on your computer to raid your retirement and empty your kids’ trust funds.

Alas, I’m not.

I am, however, truly sorry for any hardship I caused anyone as a result of my poor decisions. I regret that I didn’t do better. I desperately want to exorcise those demons. I’ve given out 6-figures worth of refunds. I’ve apologized to a lot of people. And I learned a lot of lessons the (really) hard way.

Not that I expect anyone to feel sorry for me. That would be silly. I’m the one who made the mistakes, and the accountability is mine.

That said, there are some truly sick people out there who probably need to seek professional help for the joy they feel when they witness a public crash-and-burn failure. Ruthless, despicable, small-minded, petty people. If you don’t know what I mean, try failing in the public eye sometime and see what happens. The taunting, gloating, lies, and threats. It’s sickening. Truly.

If you do ever stick your neck out and have a public failure of the epic kind, look me up. I’ll buy you a drink (or three) and we can swap stories. Turns out there’s an entire subculture in the American business economy of successful people who have been dragged through the mud (sometimes for good reason, sometimes not) and yet come back to not only fight another day, but win repeatedly and often. I’ll welcome you to the club.

Look, those 6 months were too painful for me to forget. And the aftermath held me frustratingly in its grasp for most of the ensuing 2 years. And I hated it. And I hate that I created that in my life. And, at the end, it is what it is. As the title of this post says, “C’est la vie.” It’s time for me to move on. If you hate my guts, it’s probably time for you to move on, too. If you simply can’t, and feel like you still need a piece of me, then write me a really nasty, scathing, rage-filled email and get it out of your system. I’ll most likely delete the email unread. But if it makes you feel better, then go for it.

It’s time I embrace the new me. In my life I’ve succeeded beyond what I could have reasonably expected. In my life I’ve failed beyond what I thought could be tolerable. I’ve made genius decisions and I’ve made pathetically boneheaded mistakes. Ultimately, I am who I am. And I’m a better person through all of it.

After all that’s happened, there are some mistakes I’ll never make a second time…

You’ll never catch me coaching anyone professionally again. Ever. Even though I proceeded to have a better business than ever doing exactly what I taught (and I still think it was some of the best teaching you’ll ever get), being any kind of a coach is a terrible fit for me. And that much worse for the student.

And I seriously doubt I’ll ever play big in the affiliate space again. I just don’t like the people and attitude it tends to attract. It’s a dirty, slimy, greedy game. And, unfortunately, when you screw up playing the game you’ll instantly be branded as the worst of everything bad that’s associated with it. Sad. And true. And not worth it.

And there is at least one thing I miss that I’ll definitely start doing again…

I love learning cool new stuff. And I love sharing it. I loathe feeling like I can’t share it and can’t be me. I built my (first) online reputation sharing good knowledge, powerful tools, and boring stories about my life. It was good. And I liked doing it.

Sure, the other stuff will forever be part of my internet identity. But I can’t let that keep me from doing what I do. So I’m posting to this left-for-dead blog again. And I’d like to keep doing it. Flame me if you will. Excoriate my intentions and call me names again if you must. That’s just more of the same and it’s been done to death. I’m not immune to it. It hurts. But not as much as hiding from it does.

C’est la vie. I’m back. Quietly.

Info-Marketer’s Stupid Tax…

Some people are just dumb.  Or maybe it’s just that they’re lazy.  Or possibly ignorant.

Sometimes when a new product is launching and I want to get more info about it to see if it’s something I want to promote.  Often, it’s hard to know a lot about a product with today’s vogue pre-launch page being something really mysterious and promising killer results without really telling you anything.

So awhile back I started using Google to help with my research.  I do a site search to see what pages Google knows about.  It’s a quick and easy way to see if there are affiliate pages, alternate landing / squeeze pages, special reports, etc.  It can be surprisingly effective.

By the way, if you don’t know how to do a site search on Google just type in “site:targetdomain.com” in Google and it will show you every page it has indexed.

In the process of doing this I’ve discovered that there are a *lot* of publishers out there with premium content being indexed by Google and available for the world to see and access… if they know about Google site search.

Basically all the stuff they’re selling… free to the world.  Sometimes it’s the sales thank you page… with links to all the content, of course.  Sometimes it’s a premium content signup page… which is funny.  The publisher went to the trouble of protecting the content… and left a trusted signup page exposed to Google.  Dumb.

At first I would email the site owner and let them know their content is exposed to the public.

But it’s so freaking common that I’ve stopped even doing that.

Is your content exposed?

Search your site and find out.  There are a lot of ways to protect it.  Find one and use it… or get comfortable with giving your content away for free.

And to all the info-product publishers who leave their premium content exposed to me… thank you for paying your stupid tax.

Sneaky Laser URL SEO Trick?

I like Howie Schwartz.  He does a lot of cool stuff… especially with SEO and free traffic.  And those aren’t my strong points, so I like to watch what he’s doing.  He’s smarter than me when it comes to that kind of stuff.

So I’m watching one of his free videos today.  And he’s showing off a function that I think is part of his Black Hat is Back membership… I don’t remember, exactly.

But what he showed was totally cool… And it’s something you can do with Laser URL.  Granted, it’s the poor man’s version of Howie’s automated stuff… but it works and it’s free.  If you want a more advanced tool then check out Howie’s stuff.

To me, though, every bit as important as the tool is the mindset he exposes.  I *LOVE* it.  So sneaky.  And so freaking smart.  He basically exposes a way to leech free traffic ideas from the most competitive and advanced marketers in the world.

I was seriously blown away at how simple the method it.  I’m talking stupid easy.  And obvious.  And I can’t believe I never thought of doing it or using the tools I already have (like Laser URL) to do it.

I hate getting scooped on sneaky cool ideas.  I love knowing them, though… especially when I already have a tool that lets me use them.

I had to make a few minor changes to Laser URL to get it to do what I wanted.  That means you’ll probably have to download the latest version to see what I’m seeing in the tutorial video.  You’ll need at least version 1.2.8.23.  Earlier versions will work… they just won’t have some of the features I added to make the information a little easier to work with when using this trick.  Still free.

It’s *not* totally automated like Howie’s tool is.  I might do that at some point.  I want to play around with it more.

Watch this tutorial video I just made to see what the trick is and how you can use it.  I made the video after only playing with the idea for about 15 minutes… so it’s pretty off-the-cuff.

I hope I explain it well enough in the video.  If not, please ask in the comments.  I don’t have a link to Howie’s video handy (gotta go back and check my deleted email), but if I find that again I’ll post it to the comments.

Here’s the sneaky SEO tutorial video

Sneaky SEO Trick Video

And here’s the link to download Laser URL.

Sneaky.  Sneaky.  Sneaky.

Leave a comment with your thoughts.

More on “Cheater” Landing Pages…

This post started as a reply to Bryan in my “Cat’s Out of the Bag” post.  It got long and it’s useful information so I promoted it to a post.  It’s a good look into how I use the strategy Amish exposed and, more importantly, my mindset when it comes to gimmicks and tricks in marketing.

Here’s Bryan’s original comment and my reply…

**********

Hi Matt,

If you are scraping from wikipedia relating to a keyword for your Iframe Magic page, how come google does not recognise it as duplicate content?

If a complaint was made to google by their scouts or your competition the site would inevitably be slapped.

How could you get around this Matt?

Thanks Matt

**********

@Bryan… The duplicate content scare is mostly a myth.  There are definitely ways to get penalized for trying to game google with duplicate content and syndicated sites… but most marketers will never suffer from it.  I won’t go into any more depth here… but the duplicate content penalty isn’t what you think.

The second statement is my bigger concern.  Human reviews are always a roll of the dice.  I have yet to have one of my hidden content pages slapped.  And while I can’t say exactly why that is, I do have some theories…

1. I consistently rotate content, pages and offers.  Most of this is just the nature of marketing.  I rarely have a single offer run untouched for more than several weeks at a time… usually far less.  And typically these “cheater” pages only stay up long enough for me to get to #5 below.

2. I only use public domain "scrapes" for my content.  Wikipedia, Yahoo Answers, articles, etc. are "clean" ways of getting content.  Search results are questionable, but can be effective.  I stay away from blatant site rips.

3. I routinely use content cloaking on my landing pages.  The content cloaking is more of a by-product of how I present my landing pages, though.  I optimize by keyword and rotate different content based on the search keyword.  That means my ads typically point to a "clean" landing page that is dynamically served with different content based on different user agents and sources.  It’s a little complex… and considerably more advanced than the script Amish shared.

4. I know and understand that Google at some point will probably take measures to make this less effective.  They are doing stuff like that *all* the time.  I can’t sit around wringing my hands over what they might do tomorrow… when I know something that works today.  That’s a common mistake I watch people make all the time.  Frankly, it’s one of the reasons I was irritated when Amish made the video… exposure usually leads to closure where Google is concerned.  And then I’ll move on to something else.

5. Because of #4, I don’t rely on this for a significant portion of my sites or offers.  As I mentioned in my post, I use this mostly for quickly testing markets and establishing back-end conversion ratios.  If the numbers prove hopeful (which most don’t) I "graduate" the campaign to dedicated landing pages and a more complete marketing effort.

I started doing this as an improvement on the Commission Blueprint method taught by Steve Clayton and Tim Godfrey.  The whole process led me to create the spec for Ad Grenade as a way to throw up massive amounts of campaigns very, very quickly… and without having to create landing pages for everything.

Most of those campaigns are losers.  That’s just the way it works.  I’ve learned to hate wasting my time creating entire campaigns that will simply never make money.  It’s slow, expensive and frustrating.

So instead I start off with a cheater campaign that I can get up and running with essentially a small amount of keyword research and nothing more.  I start the campaign with cheater landing pages and a basic keyword list exploded by Ad Grenade.

I really just want to see the traffic, costs and conversions for the keywords.  I may tinker with the ads… but mostly I want to know if it’s worth my time and energy to build out a complete campaign for a product.

Most keywords and cheater pages bite the dust within a week.  Those that survive typically get a complete makeover into first-class landing- and pre-sell pages.  And the cheater pages are gone there, as well.

Every once in awhile I find a keyword that I can’t improve on with my own landing pages.  On those rare occasions I’ll run that specific keyword through the cheater page long-term.  I can probably count on my fingers the number of times that’s happened.  (By the way, there’s a hint in that paragraph about an AdWords trick that gets around just about all of Google’s limitations).

Okay… does that make sense?

In a nutshell… Yes, there’s risk to doing this.  No, you shouldn’t be afraid of Google.  Yes, you should take action.  No, this isn’t a magic long-term cure all for stupid and lazy marketers.