Easy Populate and Attributes Copier time outs

[JGofGFS]

I've just set up a store that will have well over 10,000 items, uploaded about 1,500 items at a time. It's a photo site (http://regattashots.com/catalog) that consists of tons of photos each with the same ordering attributes. I did an upload this morning through Easy Populate and it went through flawlessly. I use Option Type Manager so Easy Populate won't update the text box attributes that it creates, so I was going to use Attributes Copier to copy the 10 or so attributes that each item has. This worked when I tested it on a small number of items.

Here are my problems:

Attributes Copier seems to time out when I try to copy all the attributes of one item to the 1600 items. I can do a few items (30 or 40) at a time, but that's not really a good solution for me with 10s of thousands of items.

Also..

Easy Populate won't download or save a Complete tab delimited text file. I am able to get a Model/Catagory downloaded but not a Complete list with Attributes. When i try, it just times out. I've tried different browsers too (im on Mac OS 10.3)

Has anyone experienced these problems with time outs? Or have any expereince with large stores like this? I know there are stores out there with thousands of items and it seems that Easy Populate and Attributes copier were made for these kinds of situations.

Any advice would be greatly appreciated.

Justin

[neil]

Hi Justin,
In the earlier versions of EasyPopulate there was a reason stated for this.
Maybe those larger stores are on their own servers where you can change your php.ini file- script execution time?
This is from Tim Wasson's ReadMe file from version 2.0.
As always please post what you find out here - so we all benefit.

IMPORTANT: If you have more than 300 items to upload, READ THIS:

If you have more than 300 items, there is a good chance that php
will hit the 30 second time limit for script execution as
defined in the php.ini file. Those with their own server can edit the php.ini
file to allow longer run times. Figure 30 seconds per 300 items, more or less.
Those on shared servers where php.ini cannot be changed, don't dispair.

On the Easy Populate page, in the lower file browse box, hit browse and
get your file name in there. Then click the button marked "Upload and Split File".

This will upload the large file, and split it into smaller files of 300 records each.

These new files will be saved in the /catalog/temp directory, and will be named like this:
EP_Split1.txt, EP_Split2.txt, etc. You can download these, then upload them one at a time
in the normal way. No need to manually split the file.

If these files are still to large, you can change the $maxrecs in easypopulate.php to a smaller number.
This controls how many records are put into a single file when you Split it.

[JGofGFS]

It's not in importing that I have the problem, it's exporting. I actually
have access to the php.ini file and I set the timeout to 300 and still get
the same results. It even seems to take the same amount of seconds to
timeout as it did before, so it doesn't seem to have much affect. I'll
play around with it a bit more. I'm also still having the same problems
with Attributes Copier.

I'm wondering if Attributes Copier and Sorter might be better than
Attributes Copier for this. I'm also wondering if it might be compatible
with Option Type Manager. I'll have to do a bit more research. I'm just
really trying to avoid manually dumping the data into the database.

[neil]

1. In the manual.txt
SPECIAL INSTRUCTIONS FOR MAC USERS
Because of the differences in the end-of-line characters used by Apple's operating systems, after editing and saving the tab-delimited file, you'll need to open the file in BBedit or equivalent and change the line endings to "Unix".

2. The timeout could be in php.ini :
In php.ini there are more settings to alter:
max_execution_time = 30 ; Maximum execution time of each script, in seconds
max_input_time = 60 ; Maximum amount of time each script may spend parsing request data
memory_limit = 8M ; Maximum amount of memory a script may consume (8MB)

There's also a note for Mac users in php.ini - line 501 in mine:-

; If your scripts have to deal with files from Macintosh systems,
; or you are running on a Mac and need to deal with files from
; unix or win32 systems, setting this flag will cause PHP to
; automatically detect the EOL character in those files so that
; fgets() and file() will work regardless of the source of the file.
; auto_detect_line_endings = Off

[JGofGFS]

1. The problem I am having is that I can't export the data, importing it works fine. My line breaks were Unix but that would only be important if I couldn't import. I just can't get the data out of EP. It weird because I can do a download of the whole database in the Backup section with no problem.

2. I've set these three variables to 300, 600, and 80MB respectively and still get the same behavior.

I even tried it on a PC and it hung for the same amount of time and then times out and gives me a "this page cannot be displayed error". It all seems to be related to the huge amount of attributes there are.. 10 attributes with 1-5 options for each for 1600 items. I'm up to 10 of thousands of attributes at this point. I can export in EP just fine when i click "Download Model/Category tab-delimited .txt file to edit" but when i click "Download Complete tab-delimited .txt file to edit" it just chokes and dies

[neil]

Yes,
EP is not handling the attributes in the export.
Have you tried Mozilla or maybe even Opera but that is "grasping at straws".
If I find something else out I'll post it here.

[jonthewebb]

I have had trouble using EP on large stores with many attributes. It is just not an efficient way of getting products into the db, I ended up writing my own scripts. The main problem I can see is that every single line of product has to have a value (usually null) for every attribute. when you have thousands of products and thousands of attributes, this creates an enormous file which just seems to cause problems.

There may be a solution using EP with very large stores, but I didn't find it

[neil]

Thanks Jon,
I'm guessing that http://www.hargrovescycles.co.uk/ would be an example of one of those large stores that EP was not designed for.
Is it possible to share some of those scripts - just the scripts without a ReadMe would be fine. I think I could figure it out.

[JGofGFS]

neil: I tried the opera and mozilla route. No dice. I'm at a loss at this point. I guess I'll have to examine the data see if I can write some scripts.

Jon: I agree with Neil, if you don't mind sharing, I'd love to take a look at what you've done.

[jonthewebb]

Hi

The scripts are a bit messy (as I only wrote them to use myself) and they are fairly specific to that store.... but if you send me your email addresses I'll send some files through.

The basic method is to get a text file in an exact format => run all the data into arrays (make, model, price, etc) => perform several queries on manufacturers, products and categories to get the data into the database.

jon