Python3 - Automatic robot market analyzer for price and image of Yahoo jp auction | Apple iOS Xcode YouTube Seminar #28


- With python3, collect the information of price and image
- BeautifulSoup and selenium is used for this purpose
- let me show you how it works in actual yahoo jp auction

Create robot auto market analyzer for yahoo japan auction

* Here is the highlight of YouTube tutorial.

Python can be used for web scraping and crawler for many purposes. One of the useful way is for auction market analyzer. While you sleep or out of office, your robot can works for you.

I often use it from my iPhone for my hobby. If you create this kind of html on your server, you can post some keyword to analyze it.

<form name="Form" method="POST" action="../cgi-bin/YouTube.cgi">
name: <input type="text" size="30" name="name">
<input type="submit" value="submit" name="button">

If you set some timer such as crontab, you can also use my code for automatic analyzer. Now, let me show you what kind of libraries that I used.

import logging
import time
from bs4 import BeautifulSoup
import requests
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import os.path
_dir = os.path.dirname(os.path.abspath(__file__))

This can be also used from linux server. But, in this time, let me show you how it works on mac PC. Before you do this, you need to install chrome driver with brew command.

driver = webdriver.Chrome()

except Exception as e:
return None

bs4 = BeautifulSoup(driver.page_source, "html.parser")
login_url = ""

Here is the basic step to login this yahoo site.


driver.execute_script("window.scrollTo(0, 500);")

You need to input your login ID and password with the designated variables. Your robot will type them very quickly so you need to prepare the sleep command. You can also make screenshot by the following command.


In this way, you can get the screen image named "testY.png".

After you finish your work, please don't forget the following command so that you can save your memory on your PC.


You can also download some images from yahoo jp auction with these libraries.

import requests
from bs4 import BeautifulSoup
from six.moves.urllib import request
import sys
args = sys.argv
import re
import os
import pandas as pd

This may be a little bit difficult but let me show you how it works on YouTube seminar. Please see it for the detail information.

def dl_y_images():
images = []
soup = BeautifulSoup(html,"html.parser")
for link in soup.find_all("img"):
if str(link).find('')>0:
src = link.get("src")

if ('jpg' in src or 'gif' in src or 'png' in src):

couCou = 1
for image in images:
if image.find("https")>-1:
re = requests.get(image)
re = requests.get("https:" + image)
pName = dPicName + str(couCou) + ".jpg"
with open(pName,'wb') as f:
couCou = couCou + 1

Let me add one advice. If you need the image file, you need to understand what feature the link is. Yahoo jp auction uses the following url for their images so this is the key to collect the image.

To get the source code, check the comment of my YouTube

Back to Table List