symfony
symfony forum
Members Members   Search Search   FAQ FAQ   Register Register   Login Login   Home Home
Home » support » General discussion » Load Data from a big yaml file is very slow
icon5.gif  Load Data from a big yaml file is very slow [message #38021] Wed, 24 October 2007 11:02 Go to next message
cokacola is currently offline cokacola
Messages: 34
Registered: March 2006
Location: Paris, FRANCE
Member
Hi,
I have a big problem, i want to load data from a big yaml file, 80 000 lines for 15 tables, but it's very slow, 35min by web, and indetermined by cli because i stop it after 60min.
Also I get the impression with the cli loop ???
Apparently, the first query to the database runs after 20min,by web, why is this taking so long?

If you already meet the problem or idea for optimized, I thank you in advance.


Greg

[Updated on: Wed, 24 October 2007 11:03]

Report message to a moderator
Re: Load Data from a big yaml file is very slow [message #38027 is a reply to message #38021 ] Wed, 24 October 2007 11:50 Go to previous messageGo to next message
djoos is currently offline djoos
Messages: 290
Registered: August 2007
Location: London
Faithful Member
I already met the problem as well, but have not really found a solution yet - except of just being patient Smile
Report message to a moderator
Re: Load Data from a big yaml file is very slow [message #38056 is a reply to message #38021 ] Wed, 24 October 2007 18:25 Go to previous messageGo to next message
indeyets is currently offline indeyets
Messages: 9
Registered: August 2006
Location: St. Petersburg, Russia
Junior Member
do you use syck or spyc for loading yaml?
Report message to a moderator
Re: Load Data from a big yaml file is very slow [message #38098 is a reply to message #38056 ] Thu, 25 October 2007 10:50 Go to previous messageGo to next message
cokacola is currently offline cokacola
Messages: 34
Registered: March 2006
Location: Paris, FRANCE
Member
I don't know.
I use a sfPropelData::loadData method, I suppose that it use spyc by default, But I will watch.
Otherwise, is there a real difference between the two?

thanks.
Report message to a moderator
Re: Load Data from a big yaml file is very slow [message #38100 is a reply to message #38098 ] Thu, 25 October 2007 10:55 Go to previous messageGo to next message
indeyets is currently offline indeyets
Messages: 9
Registered: August 2006
Location: St. Petersburg, Russia
Junior Member
cokacola wrote on Thu, 25 October 2007 12:50

Otherwise, is there a real difference between the two?



syck is MUCH faster
Report message to a moderator
Re: Load Data from a big yaml file is very slow [message #38101 is a reply to message #38100 ] Thu, 25 October 2007 11:03 Go to previous messageGo to next message
cokacola is currently offline cokacola
Messages: 34
Registered: March 2006
Location: Paris, FRANCE
Member
Ok
I look which is used
If it is spyc I try to integrate syck
Report message to a moderator
Re: Load Data from a big yaml file is very slow [message #38104 is a reply to message #38101 ] Thu, 25 October 2007 11:35 Go to previous messageGo to next message
cokacola is currently offline cokacola
Messages: 34
Registered: March 2006
Location: Paris, FRANCE
Member
is what Syck is available under windows?
If yes where ? please !!!
Report message to a moderator
Re: Load Data from a big yaml file is very slow [message #38105 is a reply to message #38104 ] Thu, 25 October 2007 11:38 Go to previous messageGo to next message
indeyets is currently offline indeyets
Messages: 9
Registered: August 2006
Location: St. Petersburg, Russia
Junior Member
cokacola wrote on Thu, 25 October 2007 13:35

is what Syck is available under windows?
If yes where ? please !!!


not yet, sorry Sad
I submitted some patches for that yesterday night, but more patching is still required
Report message to a moderator
Re: Load Data from a big yaml file is very slow [message #38107 is a reply to message #38021 ] Thu, 25 October 2007 11:52 Go to previous messageGo to next message
halfer is currently offline halfer
Messages: 9514
Registered: January 2006
Location: West Midlands, UK
Faithful Member
I don't think it has ever been available for Windows, unless you are prepared (and can) compile it yourself.

Remember Palestine
Report message to a moderator
Re: Load Data from a big yaml file is very slow [message #38120 is a reply to message #38107 ] Thu, 25 October 2007 15:43 Go to previous messageGo to next message
al_n is currently offline al_n
Messages: 321
Registered: May 2006
Location: The Netherlands
Faithful Member
the problem is not the YAML parser but the amount of data and the memory needed as the data loader needs to keep references of all keys.
For a large amount of data just use good old plain dump files Cool
Report message to a moderator
Re: Load Data from a big yaml file is very slow [message #38135 is a reply to message #38120 ] Thu, 25 October 2007 18:05 Go to previous messageGo to next message
cokacola is currently offline cokacola
Messages: 34
Registered: March 2006
Location: Paris, FRANCE
Member
ok
But the data in the database are crypted, and I export decrypted.
But thanks anyway Rolling Eyes
I look if I can do some things with syck on windows
Report message to a moderator
Re: Load Data from a big yaml file is very slow [message #38473 is a reply to message #38135 ] Wed, 31 October 2007 16:06 Go to previous messageGo to next message
al_n is currently offline al_n
Messages: 321
Registered: May 2006
Location: The Netherlands
Faithful Member
I re tested syck and it does make a BIG difference.
I tested with a yaml file of about 200,000 lines which I killed after an hour while using spyc. The same file was loaded with syck in about 5 minutes!!!
Report message to a moderator
Re: Load Data from a big yaml file is very slow [message #43302 is a reply to message #38021 ] Sun, 13 January 2008 01:23 Go to previous messageGo to next message
lionslair is currently offline lionslair
Messages: 89
Registered: October 2007
Location: Perth / Western Australia
Member
Did you have any issues exporting data such as blog entries and the like? How did you escape the data so as not to cause issues when reading the yml file back into the import?

www.lionslair.net.au
Report message to a moderator
Re: Load Data from a big yaml file is very slow [message #43313 is a reply to message #43302 ] Sun, 13 January 2008 11:32 Go to previous messageGo to next message
indeyets is currently offline indeyets
Messages: 9
Registered: August 2006
Location: St. Petersburg, Russia
Junior Member
lionslair wrote on Sun, 13 January 2008 03:23

Did you have any issues exporting data such as blog entries and the like? How did you escape the data so as not to cause issues when reading the yml file back into the import?


it should just work. Syck takes care about escaping
Report message to a moderator
Re: Load Data from a big yaml file is very slow [message #43314 is a reply to message #43313 ] Sun, 13 January 2008 12:05 Go to previous messageGo to next message
lionslair is currently offline lionslair
Messages: 89
Registered: October 2007
Location: Perth / Western Australia
Member
indeyets wrote on Sun, 13 January 2008 19:32

lionslair wrote on Sun, 13 January 2008 03:23

Did you have any issues exporting data such as blog entries and the like? How did you escape the data so as not to cause issues when reading the yml file back into the import?


it should just work. Syck takes care about escaping



Yes I ended up using this class. I actually copied it and re-named it so it5 stopped printing ---- when it was not a new file rather a new record. This then formatted the fields I was having trouble with then it worked.

www.lionslair.net.au
Report message to a moderator
Re: Load Data from a big yaml file is very slow [message #43324 is a reply to message #38021 ] Sun, 13 January 2008 17:51 Go to previous messageGo to next message
avatar is currently offline avatar
Messages: 14
Registered: October 2007
Location: Indonesia
Junior Member
Well i ever try that to, and post same comment just like you. Btw i 've already build some plugin to fix that problem. And add a nice progress bar to it (but sorry, i can't find it now, in my laptop maybe later i will give you that plugins).

The idea is creating task that will load csv file or an excel file, or another formatted type file. I don't find that plugin on my hardisk file now, but if you wants to build that plugins just tell me, i will glad to help you!
Report message to a moderator
Re: Load Data from a big yaml file is very slow [message #45970 is a reply to message #43324 ] Wed, 20 February 2008 19:12 Go to previous messageGo to next message
wolverina is currently offline wolverina
Messages: 1
Registered: February 2008
Location: Venezuela
Junior Member

Hello pleases me your idea and I would like to help make this a plugin. Contact me
Report message to a moderator
Re: Load Data from a big yaml file is very slow [message #46065 is a reply to message #45970 ] Thu, 21 February 2008 16:03 Go to previous message
jillelaine is currently offline jillelaine
Messages: 228
Registered: January 2008
Location: West Coast
Faithful Member
It is not a 'symfony way', but if you have MySQL as your database, and you have a large amount of data to import, you can use the LOAD DATA LOCAL INFILE with a .csv or .txt file. http://dev.mysql.com/doc/refman/5.0/en/load-data.html

I use this to import regularly from an Oracle database to MySQL. I created a file upload form where I set the parameters of the upload (fields terminated by, etc). It is VERY fast: I can import thousands of lines in only a few seconds.

But be aware, LOAD DATA INFILE does NO data validation: it will load NULL values into non-NULL columns, and blindly truncate any data that is too large for the field. Yet, if you do some data validation of your incoming file (count number of columns in each row, etc), and are prepared to find and fix bad rows, this method is very useful for large imports.
Report message to a moderator
Previous Topic:Ajax and interact with mouse clic
Next Topic:some problem with mysql serv away from my localhost
Goto Forum:
  

[ Syndicate this forum (XML) ]
powered by FUDforum - copyright ©2001-2004 FUD Forum Bulletin Board Software