Raoul, our batch (Magmi) product upload, processes the products in batches of 2000, so it appears the issue is occurring after the first batch has been completed.

Can you explain a bit about your environment?

Our infrastructure is as follows:

1. 1 varnish server + nginx for dealing with ssl encryption
2. 1 load balancer nginx
3. 2 app servers apache

Whenever a user accesses the site the request is first handled by the varnish server, if there is no cache for the specific request the request ends up in the load balancer which does a round robin to the 2 app servers. 

OK, thanks for this.

We've had a similar issue before whereby the NGinx server was terminating the connection before the request was complete and the reply could be sent back to IMan.

I suspect there's going to be a similar issue.

OK, thanks for the reply...it really helps.

We did some analysis and in the frist instance we changed the load balancer settings for client_max_body_size and proxy_read_timeout. They were set too low and the request did not get sent to the app server.

This helped the issue, but didn't resolved the problem. We then altered the varnish configuration which was also set to a too low value for the request to be completed:

initial values:

first_byte_timeout = 15s;

.connect_timeout = 30s;

.between_bytes_timeout = 30s;

The problem here was the first_byte_timeout since the app server too almost 2 minutes to return a result for processing the feed.

Voila!

Raoul, thanks for getting back to us with the issue; we'll update our documentation to include a discussion on this.