What is the undocumented web? Scott and Wes dive into it, discussing APIs, faking, scraping, automation, proxies as well as tips and tricks for best practices.

Kyle Prinsloo’s Freelancing & Beyond — Sponsor

Kyle Prinsloo teaches you everything you need to know about freelancing, including how to quit your job, earn a side-income and start taking control of your life. Check it out at https://studywebdevelopment.com/freelancing and use the coupon code “syntax” at checkout to get 25% off.

Freshbooks - Sponsor

Get a 30 day free trial of Freshbooks at freshbooks.com/syntax and put SYNTAX in the “How did you hear about us?” section.

Show Notes

3:05 - What Is the undocumented web?

When traditional methods don’t give you what you want
Wild West

6:10 - Undocumented APIS

Instagram’s private API
Use dev tools’ XHR Request
React Dev Tools will also have the data
Ep 008 - Wes Bos Origin Story
Realtor.ca - MLS Mapping
Simon Willison’s Zeit Day Talk about XML files
Charles Proxy App - also good for debugging requests

17:34 - Faking

Faking User Agents
- Sending headers from requests
- Free Blackberry wifi on plane

23:52 - Scraping

What is craping?
Cheerio - jQuery like node package

27:30 - Automation

Cron Jobs
Filling out forms instantly
Headless browsers
- Puppeteer
- PhantomJS

33:30 - Proxies

Need a fresh IP address
Need a different country
Side note: Canadians buying stuff in USA M4S1P1 === 41100
API throttling

39:00 - Tips and Tricks

Multiple API keys
diskDB - Text files as databases
Search Github for API keys
Inspect Element and Delete Overlays
Delete Cookiesokay

44:45 - Stories

Photo Voting
MLS Listing
Top Entrepreneur voting cheating
Vanity Plate Checker
My Hockey Team Sucks
Vehicle auction bidding app
youtube-dl

××× SIIIIICK ××× PIIIICKS ×××

Shameless Plugs

Tweet us your tasty treats!

Scott’s Instagram
LevelUpTutorials Instagram
Wes’ Instagram
Wes’ Twitter
Wes’ Facebook
Scott’s Twitter
Make sure to include @SyntaxFM in your tweets

The Undocumented Web: scraping, private APIs, proxies and “alternative solutions”

Syntax - Tasty Web Development Treats

The Undocumented Web: scraping, private APIs, proxies and “alternative solutions”