Web Scraping Magic…

I do a *lot* of web scraping and automation with my marketing tools.  Some of it is very advanced.  For example, I get into some pretty deep stuff with Site Sniper Pro to be able to determine the ad region location on the page.

Every once in awhile I stumble across a tool so good that I just have to share it.  Well… I’ve stumbled 😉

I usually use a program on my mac called Scoop to intercept http requests so I can see what’s happening behind the scenes of web requests.  If you’re not using something similar and you do a lot of web scraping or web automation then you’re wasting a *ton* of time.

WARNING: Geek Alert!  Ok… before I go any further I should warn you… this is not a beginner’s post.  If you don’t know what web scraping is or don’t do home-grown automation then you might want to skip this post.  It’s more technical than most of my posts.  I’m assuming you have at least a basic knowledge of what I’m talking about in the post or it won’t make sense.  You’ve been warned.  If you have questions feel free to post in the comments.

Back to the story… The 1 HUGE problem with Scoop is that it couldn’t handle secure http (https) requests very well.  This is a common problem with port scanner type software.  It gets the request too late in the game and all I can see is the encrypted stream.  Typically this is good news.  It makes it difficult for unwanted guests to eavesdrop on my browsing session.  But when I’m trying to deconstruct a complex web request it just blows.

I’ve been able to get around it in the past using an overly complex system that required entirely too much work on my part.

But recently I started a project where I wanted to create my own front-end for Google’s keyword research tool.  Their entire session is secure.  And it uses nothing but AJAX and javascript to do its work.  I was really struggling with my traditional methods to deconstruct the browser conversatio and duplicate it with my software.

After a bit of googling and trying lots of different solutions I finally found a tool that just ROCKS for what I’m trying to do.  It’s called IE Inspector (available from ieinspector.com – not an affiliate link).  It works in Windows and intercepts all major browser requests (IE, FireFox, even Chrome).  It gives me a window into the browser conversation like I’ve never had before.

Here’s short video of how I use it and what it does for me.  Enjoy.

Watch the Video...

Sneaky ClickBank Affilate Trick…

Here’s a very sneaky ClickBank trick that I’ve never seen anyone else use.  It’s not a get-rich-overnight type of trick… more like something useful to have in your toolbag for those times when it’s needed.

Here’s the Story…

On occasion I want / need to get an advance copy of a ClickBank product.  This came up to day when I started to get a flood of emails for Andrew Fox’s new Dominating ClickBank 2.  I may be able to get a preview copy from Andrew… but in this case I just didn’t have time.  It launches tomorrow and I want to preview the content to determine whether or not I can create a value-added offer for the launch.

I do this on occasion and it’s usually a quick way to make $4k – $10k.  In fact, I used this with a friend of mine recently and he made over $20,000 in the first week doing a launch product like this.

Here’s the Problem…

Affiliates are already promoting the heck out of the product.  Some of them have advance copies… most don’t.  I want a clear advantage on launch so I definitely want a copy of the product before-hand.  Waiting until launch before I see the product could cost me several thousand dollars.So how do I get an advance copy?  I could simply ask Andrew for a copy.  But that takes time and he’s probably up to his eyeballs with other launch stuff with just a day to go (even I admit I’m cutting this one close).

So I have the sneaky little ClickBank trick I use.  You see, even though his sales page isn’t even up yet his product is in the CickBank database.  And that means I can use a little bit of technical magic to go straight to the order page.

Solution, Please?

I created a nifty little software utility that finds all the possible shopping cart pages in ClickBank and lists them for me.  Then I can simply double-click on the product I want (there’s usually just 1 or 2) and order it.I’ve even had vendors sometimes email me wondering how I already bought their product.  It can be funny… one panicked because he thought he had a premature launch.  High comedy.

There’s a little risk that the vendor may not have downloads and content ready yet, but I haven’t had that problem yet.  Every time so far I’ve been able to get access to everything.  On occasion the content will be updated after I have it… but most people offer perpetual downloads these days.

Besides… worst case I just get a refund from ClickBank or I wait until the actual launch and get it then.

Anyway, this can be a very powerful (if infrequent) strategy to use to really take advantage of product launches.  You can have a gigantic edge over the poor schleps who don’t have access to the content pre-launch.

Click here to get my Sneaky ClickBank Cart Trick software.  It’s FREE.

Watch The Video…

I made a short video of the software in action so you can see what I’m talking about.  Click here to watch it.  Tricky, tricky, tricky.

Count Something.

A few days ago I started a post about about the book “Better”.  Today I’m going to finish that post.  At the end of the book was a “cheat sheet” of sorts for how to improve performance.  It was a good list.  And one of the items absolutely jumped off the page at me.  It was titled very simply: Count Something.

A few years ago when I used to work as revenue enhancement consultant for small businesses I used to really harp on the importance of knowing your numbers.  How crucial it is to business success.  And here’s what I came to learn after looking at the numbers of diving deep into the business operations of dozens of companies.

Results Never Lie

The first truth I learned is that results never lie.  That may seem ridiculously simple to you, but it’s a profoundly deep truth to really get in touch with.  Because every business owner wants to tell (and believe) a story about his/her business is where they are.  They have great reasons and excuses why right now is an aberration and tomorrow will be different.  But the results say otherwise.  Does that make any sense?  You’re far better off facing and dealing with the reality of your results than excusing them away and hoping/expecting them to get better when circumstance changes.  That probably deserves a post of it own… how people get tricked into believing that their results aren’t real, but their expectations are.

Another truth I learned is about the numbers in your business (internet or otherwise).  Numbers in business are an interesting thing.  I can look at the numbers in a business and tell you almost immediately what that business needs to work on and improve.  Now that’s another thing that when stated so plainly there’s a tendency to say “well, duh”.

So why don’t more business owners take their numbers seriously?  It’s shocking to me how few business owners even know their core numbers.  They are the pulse of our business and you can’t tell me your gross profit margin?  You can’t tell me, in internet marketing, your cost-per-visitor?  Revenue per impression?  Those are basic numbers.  What about when we get into more advanced, and more powerful, ones?

But here’s the interesting thing about numbers.  It’s also why I’m so fascinated by them.  By themselves they never give you any answers.  But they do tell you exactly which questions to ask.  And that’s powerful to know.

I think a lot of business owners feel like they don’t have the time or resources to track the numbers precisely because there are no answers in the numbers.  They think they are just for the accountants and bankers.  How wrong they are.

Knowing your numbers gives you incredible flexibility and creativity in your business.  When you know exactly which questions to ask and focus on you have freed yourself up to be ridiculously creative in answering only the questions that matter.  Most business owners spend an inordinate amount of time wandering around answerign meaningless questions their business isn’t asking.  Why?  Because they don’t know how to focus on the real questions their business has.

For example, let’s look at a typical internet business advertising on Google.  Just with the basics you can look at several levels of numbers.  Impressions, clicks, leads, and conversions are probably the most basic.  If you really look at those numbers, even their most basic and raw form they will begin to expose the questions you should be asking in your business.  If your impressions are lagging then you should be asking questions about keywords, targets and demographics, for example.  Clicks are good but conversions low?  That tells you what questions to ask.

And the human brain is so amazingly cool and powerful.  It absolutely LOVES questions.  It loves to know what to focus on.  Without the distraction of confusion you may be astounded at the answers you get when you know just what to ask.

And that (in a roundabout sort of a way ;-)) brings me to “Count Something.”

Want to make your business better continually.  Here’s an idea.  Count something.  Anything.  Here are the rules… it must be something you can continue to count.  In other words, you can’t count something that is static and doesn’t change.  And that’s the only rule.

Count Something!

And here’s what will happen over time.  Your brain will begin to ask questions about that something.  And you will start to have creative flashes about how you might influence that something.  And over time that something transforms from a external something into a malleable, pliable, shapable something that is internal to your circle of influence.

For example, there is a great story in the book about a doctor named Ingaz Semmelweis.  He started to count something in his hospital and discovered some radical answers that led to some unbelievable stunning medical improvements.  Read his story on wikipedia.  Seriously… go read it.

If you’re not counting anything right now you can start with something basic… like impressions.  Or conversions.  Or whatever.

Live that number for a period of time and you will be impressed with how much influence you begin to have over it.  You’ll be surprised at the insights you have into it.  And you may be shocked (and at times frustrated) with how many other things it shows you to count.

And when you get enoiugh of those numbers within your sphere of influence and creativity then the floodgates begin to open.  When you learn, within your own business, how to grow, shrink, or maintain a number you are learning to be master at business.

Try This…

So here’s a challenge for you.  Pick something right now in your business that you can start counting today.  In my own business (Site Sniper Pro) I might pick something like “how many of my customers are using Site Sniper Pro to sell their own products?”  Over time I can track that number and maybe gain some interesting insights in how and why people buy and use my software.

The neat thing is I don’t have to start out with answers.  I don’t know what that number is or what it means yet.  But I do know that if I keep counting that number eventually my brain will have a question about it.  And as soon as my brian starts asking questions it simultaneously starts answering them.  And just in the process of counting I can open up new avenues of business or improve my existing one.

What if I discover I have a high percentage of customer who are promoting their own products (as opposed to affiliates)?  Might that lead to changes in my sales copy?  What if it’s a low number?  Could I target them more?  Do I need to increase my affinity with that market?  You see, right now it’s a number that is entirely outside of my control because it is unknown and undiscovered.  Once I bring it into my consciousness I begin the process of influencing it and uncovering what it influences.  Makes sense?  There’s a web of intrugue there that I could follow for a long time… with constantly improving results in my business as the evidence left behind of me uncovering that web.

So count something today.  Then count it again tomorrow, or next week, or whenever it seems right.  And allow yourself to question that number.  And experiment with that number.  And learn from that number.  And your influence will be expanded.

Then count something new.


Basic Link Cloaking for Affiliates…

Over the weekend I got involved in a discussion on the Amazon.com Associates forum about the ability/need to use cloaked affiliate links.  Although the conversation there was decidely focused on the Amazon.com Associates program there were a number of good points and questions made.

This is clearly something that is not understood by many affiliate marketers.  So I think this is a good opportunity to take a few posts over the next several days and go over the upsides/downsides of and reasons for cloaking your affiliate links.

I will pay particular attention to the Amazon.com Associates program… although most of this discussion is applicable to all affiliate programs.

If you’ve got a question I haven’t answered please post a comment so everyone can see it and get the benefit of the answer.  I will not answer questions privately via email.

Also, since I’m a programmer, I have to laugh at all the affiliate cloaking software utilities available.  Please… don’t waste your money.  With very few exceptions these tools are just creating very simple http redirects for you.  You might not know what that means… but it’s nothing special and there’s no reason for you to pay for it.  Once I finish this series I’ll make a tool available for free that will do what others are charging for.

Here we go…

What is link cloaking?

At its most basic, link cloaking is just a black-hat sounding term for an http redirect.  An http redirect is just a technical sounding phrase for a link that can easily be thought of as something like a bookmark on the web.

For example, here’s an ultra-simple cloaked link:


Go ahead and type it in or click on it.  You’ll see that instead of that URL coming up in your browser you are instantly “redirected” to the homepage for my <start shameless self-promotion> excellent Site Sniper Pro software. </end shameless self-promotion>

Here’s what’s happening behind the scenes…

ssp.php is a very simple php script that does just one thing… tell the web browser that the page requested has moved.  It takes advantage of the fact that web pages move all the time on the internet.  And search engines and browsers need an easy way to be redirected to the new location.  There’s no trickery of any sort that actually happens (despite what sellers of link-cloaking scripts would have you believe).  We’re just using what’s already built into your browser for a different purpose.  If you pay attention when you’re surfing the web you’ll find these simple redirects all over the place.

Here’s the source of ssp.php:


header("Location: http://www.regnow.com/softsell/visitor.cgi?affiliate=85095&action=site&vendor=16069&ref=http://www.sitesniperpro.com");


And here’s what I’ve accomplished using this simple redirect (the easiest form of link cloaking):

  • Made that long, unreasonable URL (my affiliate URL) much more manageable and user friendly.
  • Easily hidden my affiliate ID (85095) and affiliate network (regnow.com) from basic snooping.
  • Protected myself against the first level of link swiping (or link hijacking).  This will stop about 80% of link-hijacking immediately.  In future posts I’ll get into some of the trickier forms of hijacking that will require more effort to defeat.  For the vast majority of link thieves this is all you need to stop them.  Now do yo see why you don’t need to pay $60 for a utility that does this for you?
  • Kept total control of my affiliate traffic.  For example, I wrote a credit repair guide that I give out for free on one of my websites.  In that guide I link to a handful of credit repair services for which I am an affiliate.  Since the guide is a pdf, if any of those affiliate programs ever go away (or I find a better one) I would be stuck with all those guides with an old (and potentially unprofitable) affiliate link if I had a direct link.  Now I can redirect that link through my own site and if it ever changes the links in the guide will still be good (since I only have to change the new destination on my server).  Make sense?

There are still some weaknesses with this level of link-cloaking… but it’s enough to thwart most link thefts and it gives the added benefit of being much more visually appealing.  Here are two other common ways of achieving the same result with an html page and using your .htaccess file.



<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=UTF-8">
<meta HTTP-EQUIV="REFRESH" Content="0; URL=http://www.regnow.com/softsell/visitor.cgi?affiliate=85095&action=site&vendor=16069&ref=http://www.sitesniperpro.com">




Redirect 301 /ssp.html http://www.regnow.com/softsell/visitor.cgi?affiliate=85095&action=site&vendor=16069&ref=http://www.sitesniperpro.com


As you can see, basic link-cloaking is very easy to do.  I cloak almost all of my affiliate links with at least basic cloaking.  In future posts I’ll get into more detail about why you would or wouldn’t cloak, how your links and cookies can (and do) get hijacked, and give you some tools to do your own basic and advanced cloaking.

Got questions (even though I already know I haven’t covered everything)?  Post a comment and I’ll try and get them all answered.  Inflammatory and spam comments will not be moderated.