Need an expert developer for the development a file scraper to scrape device firmware files and their info from various vendors websites

Profile

Need an expert developer for the development a file scraper to scrape device firmware files and their info from various vendors websites
A python based CLI script that can download all product’s firmware (including all versions) from web pages for a given list of predefined vendors and store the information (meta data) in SQLite [login to view URL] mandatory metadata fields include ( Manufacturer, Model, Version, Type, Name, Release Date(if available), Download link, ( calculated Sha2 hash of the file)i.e. ( Cisco, Video Surveillance 6030 IP Camera, 2.7.0, IP Camera, [login to view URL], 21/08/2015, link ) There is a non-mandatory binary field which indicates if the device is discontinued or not depending on the fact that vendor mention that on the website or not. The firmware files itself will be stored in the file system and will be referenced by index ID in SQLite.
The arguments to the script should be a list of comma separated vendor names or the location of a text file containing the vendor name.
There are no GUI components in the server where the script will run so headless mode for browser should be used
Solution Scope
1. Script will be written per vendor. This is required because each vendor website will have its own implementation of the firmware download page.
2. The script will only download new firmware that have been added by the vendor. Hence first execution of script will download all the firmware available but the subsequent runs will only download new ones which will get added. This will be achieved by analysing data available in SQLite and skipping the files that are already been downloaded and processed.
3. Each vendor, that will be provided, will be analysed manually to identify the following, which will be required to develop the script:
a. URL for the firmware download page
b. Credential Requirements (Simple Signups, Specific Signups, No Signups)
c. Any Captcha on the page
d. Any honeypot traps
4. If there are credential required to download the firmware and the credentials are simple ones where a simple sign up is required, the signup will be done manually as part of the manual analysis using a gmail account dedicated for this work.
5. Script will try to imitate human like behaviour (to a limit) while scraping the web page as well as uses Tor, so that if the vendor site has scraper/crawler detection logic implemented, it can be skipped. This will be achieved by adding random delays, random view time, avoiding honeypot traps through manual analysis
Solution Brief
A Python Selenium (if required, sometimes simple requests do the job) and SQLite based solution will be developed which will have the following features/components:
1. File Management Module: Responsible for storing and managing the downloaded files and meta data. Firmware and installer files will be stored on the filesystem which will have a structured folder hierarchy. Meta data of the files will be stored in SQLite. Meta Data will refer to the stored files through paths on the file system and file index/name.
2. Vendor Scrappers: Python Selenium based scrapper will be written for each of the vendor, responsible for downloading the files and grabbing the meta data from the vendor’s site. This will make use of the file management module to store the file and meta data to SQLite.
3. Configuration File: All the configurations for the framework (including vendor specific like credentials, url etc) will be stored in a json file which can be easily modified.
4. Execution Script: The configuration file can be setup to represent the polling interval for each of the vendor scraper and when the execution script is run it will go and schedule each of the vendor scripts individually according the polling interval defined in the config.
We already developed the main skeleton of the scraper including some scrapers and need someone to develop new scrapers for each vendor. There are around 100 vendors and the milestones are defined per vendor and each milestone is max 50€ which is paid after we test the scraper and see no errors. The developer MUST test the scraper before delivering it to us.
Skills:
Web Scraping
, Selenium
, Python
, SQLite
, Linux
About the Employer:
( 3 reviews )
Brussels, Belgium
Project ID:
#25292249
Offer to work on this job now! Bidding closes in 6 days
Open - 6 days left
Your bid for this job
EUR
Your email address
Bid on this job
Set your budget and timeframe
Outline your proposal
Get paid for your work
Its free to sign up and bid on jobs
23 freelancers are bidding on average €3989 for this job
stevobujica91
Hi,
I am an expert Python Developer and i have enough expertise on scraping projects. (you can check reviews on my profile)
I can start your project immediately. we will discuss details over chat.
Best Regards.
€5000 EUR
in 21 days
(46 Reviews)
5.8
songhku925
Hi, Dear.
How are you?
I have read your long description carefully, and I understood what you want.
I agree with you about all the requirements.
As you can see in my profile and reviews, I have good experience with Pyt
More
€4800 EUR
in 50 days
(43 Reviews)
6.0
mmadi
Hi Client, checked (Need an expert developer for the development a file scraper to scrape device firmware files and their info from various vendors websites) I have strong experience in Linux, Python, Selenium, S
More
€3500 EUR
in 36 days
(9 Reviews)
6.0
george1205
Hi, I read your requirements specifically.
I am a Python web scraping expert and have a lot of experience with python selenium.
You can easily see it if you go to my Profile page.
So Im pretty sure I can build your pr
More
€3800 EUR
in 15 days
(6 Reviews)
5.4
mostshalaby
Hi.
Having a great experience in the field of Web Scraping, I can implement your project successfully by using Python(BeautifulSoup or Selenium, Pandas).
Looking forward to your kind reply. Nice regards.
Swiir.
€3000 EUR
in 7 days
(9 Reviews)
5.5
vovakovalkov80
[login to view URL] I am very interested in your project.
I am confident of this project as Im a professional Python/C#/Selenium/CEFSharp expert with over 10 years of experience.
Please feel free to contact me, to get the perfect
More
€4000 EUR
in 7 days
(1 Review)
4.2
drishinfotech
Hello I have read the job description and can develop a scraping script which will scrape the data as per given instructions. I have all the required skills you are looking for. I have around 5 years of experience i
More
€4500 EUR
in 7 days
(11 Reviews)
4.8
benliao
Hello, sir. How are you?
I have just read your project description and I can help you as my rich experiences.
I have rich experiences in these parts
I will give you the perfect result you are satisfied with.
My client
More
€4000 EUR
in 40 days
(5 Reviews)
3.8
ClementFes
Hi, Dear I have read your description carefully and convinced it is the job just for me.
As you can see on my profile, I have experience of scrapping info from several sites on real time.
⭐ I use python(selenium, beau
More
€3000 EUR
in 20 days
(3 Reviews)
2.7
YankoAlexandrov
Hello,
We are the one-stop provider for all your Scraping, Data Extraction, and Web Automation needs. Get a complete end-to-end solution on time and budget with full integration support.
I have 7 years of experience a
More
€3500 EUR
in 7 days
(2 Reviews)
2.6
MagicScripter
I will write the python scripts requsted to scrape the vendor websites. You mentioned that each websites scraper would have to be written uniquely to fit. I will also be able to work with SQLite. I believe we can work
More
€4000 EUR
in 14 days
(3 Reviews)
1.4
Dailer11
Dear Sir. Thank you for your job posting.
I have tons of experience in python scrapping with selenium. It will be no problem to do your job with my skills. If you want I can share my previous work on my PC remotely.
Pl
More
€3999 EUR
in 15 days
(1 Review)
0.7
gargankit642
Nice to meet you
I am an Amazon Cloud Architect for the web infrastructure serving 90 million page impressions and 12 TB Internet traffic per month. The AWS services I use are EC2, ELB, MySQL RDS, VPC, CloudFront, Elas
More
€4649 EUR
in 66 days
(0 Reviews)
0.0
zelishas45
Hello, Thank You For Job Posting :) I have 6 years of experience in Website and Application development. I have checked The requirement and understand that as well. I have created SIMILAR in past. Do you want to
More
€4000 EUR
in 7 days
(0 Reviews)
0.0
Soniyakumar
Hey Hello There, It’s good to see a serious job post here. This is how every client should post his job crisp & clear. Our brief introduction, We are a team of Working Professionals having around 8 years of experien
More
€4000 EUR
in 7 days
(0 Reviews)
0.0
Rogerfalcone
Hi,dear. I am very interested in your project - Need an expert developer for the development a file scraper to scrape device firmware files and their info from various vendors websites. Ive already done this kind of
More
€4444 EUR
in 15 days
(0 Reviews)
0.0
tgltech
Hi, I have gone through your requirements Need an expert developer for the development a file scraper to scrape device firmware files and their info from various vendors websites and can absolutely assist you in a
More
€3560 EUR
in 7 days
(1 Review)
0.0
Hrtechllc
Greetings, We are a company working for the last 5 years, but we are new to freelancer.com, so you might find our profile new. Our reviews do not display the amount of work that we have done, so you can go through our
More
€4000 EUR
in 40 days
(0 Reviews)
0.0
ak27freelancer
Hello,
I am pleasure with your job for Need an expert developer for the development a file scraper to scrape device firmware files and their info from various vendors websites.
Thank you for the job posting. It’s a ple
More
€4000 EUR
in 7 days
(0 Reviews)
0.0
sd0401
Hello there, We are a team of app/web designer and developer and we work on PHP, Codeigniter MVC, Laravel, Android SDK, MySQL, Xcode IDE(Objective-C), Swift, Eclipse IDE, and SQL server database, UI design, Android st
More
€4000 EUR
in 7 days
(0 Reviews)
0.0
Offer to work on this job now! Bidding closes in 6 days
Open - 6 days left
Your bid for this job
EUR
Your email address
Bid on this job
Set your budget and timeframe
Outline your proposal
Get paid for your work
Its free to sign up and bid on jobs
Post a project like this
Other jobs from this employer
Project for Majid (€30-250 EUR)
Need an expert developer to develop a Django React based client/server webapplication (€1500-3000 EUR)
Need an expert to develop a file scraper to scrape device firmware files and their info from various vendors websites (€3000-5000 EUR)
Need an expert to develop a file scraper to scrape device firmware files and their info from various vendors websites -- 2 (€3000-5000 EUR)
Developing a file scraper to scrape firmware files and their info from various vendors websites (€12-18 EUR / hour)
Next Job >
Similar jobs
installation of a componente (€8-30 EUR)
Script download from hotstar ($30-250 CAD)
EXTRACT DATA FROM PDF TO EXCEL -- 2 ($15-25 USD / hour)
Teltonika Router DUAL SIM Configuration Troubleshooting -- 2 ($30-250 USD)
Ethical Hacking Support ($15-25 USD / hour)
gui in tkinter , odoo api and mobile app (₹12500-37500 INR)
Install and Configure Social Media platform ($750-1500 USD)
[COD:AUT2] Simple automation for website ($10-30 USD)
python within scripting ($10-30 USD)
Python Expert needed ($250-750 USD)
Software app (€30-250 EUR)
AWS Server Admin ($8-15 USD / hour)
Get list of instagram followers ($10-30 USD)
Operational Mines in Queensland Australia ($30-250 AUD)
New VPS on Contabo, need to setup Plesk CP, ftp user on windows server, sql database user, roles (£20-250 GBP)
estimate the pendulums motion ($10-30 USD)
Need a AWS Cloud/Linux Administrator (₹37500-75000 INR)
Zabbix Expert ($8-15 SGD / hour)
server bind :creation 4 machine vm on vmware exy6.7 and install script ($10-30 USD)
Crypto expert required.. -- 2 ($10-100 AUD)
Need to hire a freelancer for a job?
Its free to sign up, type in what you need & receive free quotes in seconds
Enter your project description here
Post a Project
English - US (International)
Help & Support
Freelancer
Categories
Projects
Contests
Freelancers
Enterprise
Preferred Freelancer Program
Project Management
Local Jobs
Showcase
API for Developers
About
About us
How it Works
Security
Investor
Sitemap
Quotes
News
Terms
Privacy Policy
Terms and Conditions
Copyright Policy
Code of Conduct
Fees and Charges
Apps
Registered Users
Total Jobs Posted
Freelancer ® is a registered Trademark of **MEMBERS ONLY**SIGN UP NOW***. (ACN 142 189 759)
Copyright © 2020 **MEMBERS ONLY**SIGN UP NOW***. (ACN 142 189 759)
×
Welcome,
Processing …
Welcome
Link to existing Freelancer account
The email address
is already associated with a Freelancer account.
Enter your password below to link accounts:
Username:
Password:
Link Accounts
Close
I am a new user
I am a returning user
Link your account to a new Freelancer account
Email address
Email address
Username
Valid username
I am looking to
Hire
Work
Create Account
Link to your existing Freelancer account
Username or Email
Password
I forgot my password
Link Accounts

Company info

Sign Up Now - EmploymentCrossing.com