Tools that do not have "Run Python script in process" checked run extremely slowly under ArcGIS 9.3

[jjr8]

These tools will generate the warning:

This tool may run slowly due to a geoprocessing performance degradation that ESRI introduced in ArcGIS 9.3. We are following up with ESRI about this problem and will try to get them to fix it. Please contact us if you have any questions. In the mean time, you can avoid this problem by running this tool on a computer with ArcGIS 9.1 or 9.2.

The problem is that ESRI degraded the performance of geoprocessing in 9.3 for out-of-process tools. In 9.1 and 9.2, all scripts ran out-of-process. In 9.3, ESRI introduced the in-process option. In-process tools in 9.3 run much faster than out-of-process tools in 9.1. or 9.2, so this option was a good improvement. But at the same time, ESRI appears to have degraded out-of-process performance.

For more information, see the thread I opened at the ESRI forums:  http://forums.esri.com/Thread.asp?c=93&f=1729&t=264281&mc=0

There is no immediately obvious solution to this problem. I will follow up with ESRI about it. Based on past experience, I'm pretty sure they will refuse to fix it, in which case I'm not sure what I will do. The MGET scripts that run out-of-process do so with very good reason: they will not run in-process without making ArcCatalog or ArcMap unstable or otherwise breaking something. I will restructure as many as I can to run as in-process, but some will not be able to. If you have questions, please contact me.

You can avoid this problem by using Arc 9.1 or 9.2.

[jjr8]

I heard back from Ralf Gottschalk, ESRI Development Technical Lead - SDK Unit, about this problem. He said:

"NIM044619 – The issue related to out of process scripts running slower at 9.3 than 9.2. This issue is not going to be fixed at this time. I know this is not good news for you but maybe an explanation might help. The current architecture of python scripts in ArcGIS is using a named pipe. When a script is run out of processes, ArcGIS will actively listen for at it at some interval. At 9.1 we had it constantly listening for the python script, which caused it to look like it was pegging the CPU (and the reason for NIM004894). At 9.3, they introduced running scripts in processes, and at the same time changed how frequently ArcGIS listens for out of processes scripts, (less frequently in fact) so the CPU would not be pegged as heavily. As a result, the scripts run slower. This was a conscious decision and was made at the expense of performance because 1) we have in processes and 2) doing it the other way is not good either and it gives the false impression as to what is actually happening to the script.

So the story is, ESRI wants everyone to run their scripts in processes. We also know that some people run their scripts out of processes because they have to use of 3rd party dlls, and some of these dlls are not thread safe and can cause conflicts with ArcGIS. Our recommendation to people is to try to run the script in processes, and if you have problems, then run it out of processes. Have you verified that you run into problems running scripts in process? If you can try running in processes, and if you have problems then run them out of processes.

There may be a better option to how we run python scripts instead of using the named pipe, but at this time this is what we have. When you come out for holistic testing, the developers whom I talked to said they would be willing to have a discussion with you their current design and listen to any input you may have."

That is pretty much the design I had deduced :-(

Ralf invited me to go out there for "Holistic Testing". This will probably happen in late July or early August. I'll be able to interact directly with developers on the geoprocessing team. I will try very hard to convince them that there are some easy solutions to this that give decent performance without requiring a complete redesign (e.g. polling using a dynamically changing interval).

To work around this problem, I modified MGET's internal copy of the rpy Python package to allow it to be used from in-process tools, and changed all of the MGET Statistics tools to run in process. Now, the only MGET tools that do not run in process are those that invoke MATLAB code. At present, these are only the coral reef connectivity tools. In MGET 0.8, I will probably introduce the eddy-detection tools, which also use MATLAB.

The workaround was implemented in [404], and released in MGET 0.7b1.