Hi all,
I'm very new to php, but I'm learning as time goes along. I'm currently making a webcrawler in php for my a level computer science project. I have it working, getting all the links off of a page, trying to run the page, and then if it doesn't throw and error whilst requesting the content of the next page, it loops around, I am currently just looking to see what links work and what links won't work and if there's anything I can do to make them work so I can get more websites into my index.
My main issue currently is at about 15000 websites in (which takes like an hour and a half to get to which is very time consuming) I get a content encoding error.
Is there anyway to fix this so I don't get an error every time?
Thanks in advance
Here is my code for those that would like it for help (it's just an amalgamation of other people's code along with some logic based stuff on my end, I turned off warnings because id get about 600 errors come through from just trying to load wikipedia and it was majorly slowing down my already slow code):
<?php
error_reporting(E_ALL ^ E_WARNING);
$currenttime = time();
// define the target URL
$StartUrl = 'https://en.wikipedia.org/wiki/Main_Page';
/**
* Get a web file ( HTML, XHTML, XML, image, etc. ) from a URL. Return an
* array containing the HTTP server response header fields and content.
*/
function get_web_page( $url ) {
$user_agent = 'Computer Science project Web Crawler';
$options = array(
CURLOPT_CUSTOMREQUEST =>"GET", //set request type post or get
CURLOPT_POST =>false, //set to GET
CURLOPT_USERAGENT => $user_agent, //set user agent
CURLOPT_COOKIEFILE =>"cookie.txt", //set cookie file
CURLOPT_COOKIEJAR =>"cookie.txt", //set cookie jar
CURLOPT_RETURNTRANSFER => true, // return web page
CURLOPT_HEADER => false, // don't return headers
CURLOPT_FOLLOWLOCATION => true, // follow redirects
CURLOPT_ENCODING => "", // handle all encodings
CURLOPT_AUTOREFERER => true, // set referer on redirect
CURLOPT_CONNECTTIMEOUT => 120, // timeout on connect
CURLOPT_TIMEOUT => 120, // timeout on response
CURLOPT_MAXREDIRS => 10, // stop after 10 redirects
//this is very poor technique but it makes the code work
CURLOPT_SSL_VERIFYPEER => false, // disables their verifying certificate
//CURLOPT_SSL_VERIFYHOST => false, // disables my verifying certificates
);
$ch = curl_init( $url );
curl_setopt_array( $ch, $options );
$content = curl_exec( $ch );
$err = curl_errno( $ch );
$errmsg = curl_error( $ch );
$header = curl_getinfo( $ch );
curl_close( $ch );
$header['errno'] = $err;
$header['errmsg'] = $errmsg;
$header['content'] = $content;
return $header;
};
//$array = get_web_page($StartUrl);
//echo $array['errno'].'<br>';
//echo $array['errmsg'].'<br>';
//echo $array['content'];
$stackdepth = 1;
$maxstackdepth = 3;
$cfs = 1;
$pagecount = 0;
$arraylength = 1;
$worked = 0;
$workings= array();
$fails = array();
$banned = array();
$file = "webcrawllinklist.txt";
$txt = fopen($file, "w") or die("Unable to open file!");
$linksArray = array();
array_push($linksArray,$StartUrl);
array_push($banned,$StartUrl);
while ($arraylength !=0){
$array = get_web_page($linksArray[0]);
$pagecount ++;
echo $pagecount."<br>";
//echo $array['errno'].'<br>';
//echo $array['errmsg'].'<br>';
if($array['errno'] == 0){
$worked ++;
array_push($workings, $linksArray[0]);
$Page = new domDocument;
$Page -> loadHTML("<html>".$array['content']."</html>");
$Page->preserveWhiteSpace = false;
$links = $Page->getElementsByTagName('a');
$links = iterator_to_array($links);
for($i = 0; $i < count($links);$i++) {
$link = $links[$i];
$link = $link->getAttribute('href' );
//echo $link."<br>";
$if = strpos( $link, "//" );
$start = $if == false? false: $if+2;
//echo $start."<br>";
if ( $start ) {
$link = substr( $link, $start, strlen( $link )-( $start ) );
}
;
//echo $link."<br>";
if ($stackdepth != $maxstackdepth){
if (!in_array($link,$banned)){
array_push( $linksArray, $link);
array_push($banned,$link);
$arraylength ++;
};
};
};
} else{
array_push($fails,$linksArray[0]);
};
array_splice($linksArray,0,1);
$arraylength --;
$cfs--;
if ($cfs == 0){
$stackdepth++;
$cfs += $arraylength;
};
};
$output = "Works:\n";
for ($i=0;$i<count($workings);$i++){
$output = $output.$workings[$i]."\n" ;
};
$output = $output."\n Failed: \n";
for ($i=0;$i<count($fails);$i++){
$output = $output.$fails[$i]."\n" ;
};
echo $pagecount."= num of pages <br>";
echo $worked."= num that worked <br>";
$newtime= time();
$timeelapsed = $newtime - $currenttime;
$output = $output."\n time taken: ".$timeelasped;
echo $timeelapsed."= time taken <br> <br><br><br><br><br>";
fwrite($txt, $output);
fclose($txt);
header('Content-Description: File Transfer');
header('Content-Disposition: attachment; filename='.basename($file));
header('Expires: 0');
header('Cache-Control: must-revalidate');
header('Pragma: public');
header('Content-Length: ' . filesize($file));
header("Content-Type: text/plain");
readfile($file);
?>
The error on Firefox specifically says "content encoding error
The page you are trying to view cannot be shown because it uses an invalid or unsupported form of compression."
I'm using MAMP on an emulated windows 11 (I'm running Linux but I couldn't get any other local hosting apps to work so I'm using one my computer science teacher showed me on an emulated windows 11 because my school laptop doesn't let CURL work)