Bu kod nasıl daha verimli hale getirebilir, 3.5 milyon satır üzerinde döngü gerekiyor?

0 Cevap php

I have a csv file that has 3.5 million codes in it.
I should point out that this is only EVER going to be this once.

Csv gibi görünüyor

age9tlg,  
rigfh34,  
...

İşte benim kod:

ini_set('max_execution_time', 600);
ini_set("memory_limit", "512M");
$file_handle = fopen("Weekly.csv", "r");
while (!feof($file_handle)) {
    $line_of_text = fgetcsv($file_handle);

    if (is_array($line_of_text))
        foreach ($line_of_text as $col) {
            if (!empty($col)) {
                mysql_query("insert into `action_6_weekly` Values('$col', '')") or die(mysql_error());
            }
    } else {
        if (!empty($line_of_text)) {
            mysql_query("insert into `action_6_weekly` Values('$line_of_text', '')") or die(mysql_error());
        }
    }
}
fclose($file_handle);

Is this code going to die part way through on me? Will my memory and max execution time be high enough?

NB: This code will be run on my localhost, and the database is on the same PC, so latency is not an issue.


Update:
here is another possible implementation. This one does it in bulk inserts of 2000 records

$file_handle = fopen("Weekly.csv", "r");
$i = 0;
$vals = array();
while (!feof($file_handle)) {
    $line_of_text = fgetcsv($file_handle);

    if (is_array($line_of_text))
        foreach ($line_of_text as $col) {
            if (!empty($col)) {
                if ($i < 2000) {
                    $vals[] = "('$col', '')";
                    $i++;
                } else {
                    $vals = implode(', ', $vals);
                    mysql_query("insert into `action_6_weekly` Values $vals") or die(mysql_error());
                    $vals = array();
                    $i = 0;
                }
            }
        } else {
        if (!empty($line_of_text)) {
            if ($i < 2000) {
                $vals[] = "('$line_of_text', '')";
                $i++;
            } else {
                $vals = implode(', ', $vals);
                mysql_query("insert into `action_6_weekly` Values $vals") or die(mysql_error());
                $vals = array();
                $i = 0;
            }
        }
    }
}
fclose($file_handle);

ben bu yöntemi kullanmak için ise en yüksek değeri ne i seferde eklemek için ayarlayın?


Update 2
so, ive found i can use

LOAD DATA LOCAL INFILE  'C:\\xampp\\htdocs\\weekly.csv' INTO TABLE  `action_6_weekly` FIELDS TERMINATED BY  ';' ENCLOSED BY  '"' ESCAPED BY  '\\' LINES TERMINATED BY  ','(`code`)

but the issue now is that, i was wrong about the csv format, it is actually 4 codes and then a line break, so fhroflg,qporlfg,vcalpfx,rplfigc,
vapworf,flofigx,apqoeei,clxosrc,
...

so i need to be able to specify two LINES TERMINATED BY
this question has been branched out to Here.


Update 3
Setting it to do bulk inserts of 20k rows, using

while (!feof($file_handle)) {
   $val[] = fgetcsv($file_handle);
   $i++;
   if($i == 20000) {
      //do insert
      //set $i = 0;
      //$val = array();
   }
}

//do insert(for last few rows that dont reach 20k

but it dies at this point because for some reason $val contains 75k rows, and idea why?
note the above code is simplified.

0 Cevap