__color__	__group__	ticket	summary	component	version	type	owner	status	created	_changetime	_description	_reporter
5	0.8 Release	534	With ArcGIS 10, calling GeoEco tools from a Python geoprocessing script  fails with SyntaxError: EOL while scanning string literal	Core - ArcGIS Interop		Defect	jjr8	new	2012-03-26T22:05:28-0400	2012-03-26T22:05:28-0400	"The way it is supposed to work is that you need to call {{{arcpy.ImportToolbox}}} on the MGET toolbox before calling the functions. Unfortunately that is currently broken:

{{{
#!python
>>> import arcpy
>>> arcpy.ImportToolbox(r'C:\Program Files\GeoEco\ArcGISToolbox\Marine Geospatial Ecology Tools.tbx', 'GeoEco')
Traceback (most recent call last):
  File ""<stdin>"", line 1, in <module>
  File ""C:\Program Files (x86)\ArcGIS\Desktop10.0\arcpy\arcpy\__init__.py"", line 94, in ImportToolbox
    return import_toolbox(input_file, module_name)
  File ""C:\Program Files (x86)\ArcGIS\Desktop10.0\arcpy\arcpy\toolbox_code.py"", line 411, in import_toolbox
    mymodule = generate_toolbox_module(toolbox, None, False, False, False, module_name)
  File ""C:\Program Files (x86)\ArcGIS\Desktop10.0\arcpy\arcpy\toolbox_code.py"", line 389, in generate_toolbox_module
    mycode = compile(code.encode('utf-8'), toolbox.pathName, 'exec')
  File ""C:\Program Files\GeoEco\ArcGISToolbox\Marine Geospatial Ecology Tools.tbx"", line 34442
    very small discrepancies compared with the navigation-corrected version.""""""""
                                                                               ^
SyntaxError: EOL while scanning string literal
>>>
}}}

I’m not sure what the problem is. I have been testing MGET with Python but I have been using the 9.x style of geoprocessing. This still works with Arc 10:

{{{
>>> import arcgisscripting
>>> gp = arcgisscripting.create(10.0)
>>> gp.AddToolbox(r'C:\Program Files\GeoEco\ArcGISToolbox\Marine Geospatial Ecology Tools.tbx')
>>> gp.ArcGISLinesFromVectorComponentRasters_GeoEco(r'C:\Temp\u_Layer1.img', r'C:\Temp\v_Layer1.img', r'C:\Temp\test_vectors4.shp', 0.2)
Creating lines in C:\Temp\test_vectors4.shp from vector component rasters C:\Temp\u_Layer1.img and C:\Temp\v_Layer1.img.
<geoprocessing server result object object at 0x13CFD890>
>>>
}}}

As far as I have been able to tell, it is generally safe to use both {{{import arcgisscripting}}} and {{{import arcpy}}} from the same Python program. arcpy itself is a pure Python module that wraps arcgisscripting."	jjr8
6	0.8 Release	389	Find and Convert ArcGIS Rasters to Polygons fails with ValueError: The feature class ... has the shape type 'polyline', but this function requires a feature class with the shape type 'polygon'	Tools - Conversion		Defect	jjr8	assigned	2009-10-12T08:43:14-0400	2010-01-06T15:00:25-0500	"ArcGIS geoprocessing output is similar to this:

{{{
Executing (Find and Convert ArcGIS Rasters to Polygons): ArcGISRasterFindAndConvertToPolygons H:\Sea_Ice\UBremen_Ice_Concentration_Rasters H:\Sea_Ice\UBremen_Ice_Edge_Shapefiles false # * false # # # # # # # # # os.path.join(outputWorkspace, 'Ice_Edge_' + os.path.basename(inputRaster)[4:] + '.shp') os.path false H:\Sea_Ice\UBremen_Ice_Edge_Shapefiles
Start Time: Thu Oct 08 14:40:35 2009
Running script ArcGISRasterFindAndConvertToPolygons...
Finding ArcGIS rasters and inserting rows into table ""work"": workspace=""H:\Sea_Ice\UBremen_Ice_Concentration_Rasters"", wildcard=""*"", searchTree=False, rasterType=""None""
Finished inserting rows: 0:00:00 elapsed, 124 rows inserted, 0:00:00.003798 per row.
Updating field ""outputFeatureClass"" in table ""work""...
Update complete: 0:00:00 elapsed, 124 rows updated, 0 deleted, 0 unchanged, 0:00:00.001330 per row.
Querying the table ""work"" to build lists of inputs and outputs...
Query complete: 0:00:00 elapsed, 124 rows retrieved, 0:00:00.000862 per row.
ValueError: The feature class H:\Sea_Ice\UBremen_Ice_Edge_Shapefiles\Ice_Edge_20061201.shp, specified for the element 0 of the outputFeatureClassList parameter (where 0 is the first element), has the shape type 'polyline', but this function requires a feature class with the shape type 'polygon'. Please provide a feature class with shape type 'polygon'.
<type 'exceptions.ValueError'>: The feature class H:\Sea_Ice\UBremen_Ice_Edge_Shapefiles\Ice_Edge_20061201.shp, specified for the element 0 of the outputFeatureClassList parameter (where 0 is the first element), has the shape type 'polyline', but this function requires a feature class with the shape type 'polygon'. Please provide a feature class with shape type 'polygon'.
Failed to execute (Find and Convert ArcGIS Rasters to Polygons).
End Time: Thu Oct 08 14:41:03 2009 (Elapsed Time: 28.00 seconds)

}}}

This problem occurs when an output feature class exists but has the wrong shape type. The expected result is that the tool would overwrite the existing output (if overwrite is turned on) or fail with a message saying that the output feature class already exists. Instead the tool is failing to overwrite the output feature class, and instead complaining that it has the wrong shape type.

The workaround to this is to delete the existing feature class manually before running the tool.

If the existing feature class is the expected shape type, then the tool functions properly (it either overwrites it or fails saying it already exists)."	jjr8
6	0.8 Release	402	Add ConnMod Tools	Unknown		Task	bbest	new	2010-03-30T20:34:09-0400	2010-05-06T15:37:21-0400	"Add tools from the updated [http://www.nicholas.duke.edu/geospatial/software Connectivity Modeling (ConnMod) toolkit]:

 * Analysis
   * Create Network
   * Truncate Network
   * Network Least Cost Paths
   * Network Centrality Metrics
 * Display
   * Display Paths by Selected Edges
   * Display Paths by Selected Nodes"	bbest
6	0.8 Release	416	Create Table from ESRL Climate Index Time Series tools does not recognize file geodatabases	Tools - Data Products - NOAA ESRL		Defect	jjr8	new	2010-08-13T17:37:11-0400	2010-09-09T20:51:13-0400	The tools try to create a .dbf file inside the !BlahBlah.gdb directory. (This is what happened on ArcGIS 10, at least.)	jjr8
6	0.8 Release	457	Convert ArcGIS Raster to Lines tool fails if all cells of the input raster are No Data	Tools - Conversion		Defect	jjr8	new	2011-01-30T18:35:22-0500	2011-01-30T18:35:22-0500	"The problem is in the ArcGIS Raster to Polyline tool; this tool creates an empty feature class (which is good) but then reports errors and raises an exception that contains the text ""empty feature class"".

In order to work around this, we have to log errors as warnings and then ignore the exception if it contains the text ""empty feature class""."	jjr8
6	0.8 Release	461	Find Nearest Features tool should allow the user to specify which projected coordinate system to use when both the points and the near features are projected but in different coordinate systems	Tools - Spatial Analysis		Enhancement	jjr8	assigned	2011-02-06T16:03:38-0500	2011-02-10T12:07:55-0500	Currently, in that situation, the tool always projects the points to the near features' coordinate system. That may not be what the user wants. For example, I used the Cayula-Cornillon tool to identify fronts in HYCOM SST, which uses a Mercator coordinate system. I have points that are in an Albers coordinate system which is configured to minimize overall spatial distortion. I would like to project the fronts (which I converted to polylines) to the Albers system before the distance is calculated.	jjr8
6	0.8 Release	470	Various MGET tools fail with ValueError: The value specified for the element XXXXX of the XXXXX parameter (where 0 is the first element), XXXXX, exists but it is not a raster or raster layer. Please specify a raster or raster layer.	Core - ArcGIS Interop		Defect	jjr8	new	2011-03-10T10:58:37-0500	2011-03-10T10:58:37-0500		jjr8
6	0.8 Release	472	"When an invalid username or password is provided, Aviso tools fail with ""Failed to open URL ... with OPeNDAP due to Exception: Unable to parse token: <!DOCTYPE"""	Tools - Data Products - Aviso		Defect	jjr8	new	2011-04-13T11:56:53-0400	2011-04-13T11:56:53-0400	"The problem is that when the server returns an HTTP 401 Authorization Required, pydap cannot recognized the returned HTML. We should see whether it makes sense to special-case this problem and have pydap or the Aviso code raise a descriptive error, since it is likely to be a common among MGET users.

If you encounter this problem, the solution is to provide a correct username and password. As of April 2011, Aviso apparently still had not enabled their OPeNDAP server to work with personal usernames and passwords that Aviso issues. Instead, the only credential that would work was:

Username: aviso-users[[BR]]
Password: grid2010

If you cannot get your personal credential to work, try the one above."	jjr8
6	0.8 Release	474	For Aviso datasets in geographic projections (Mediterranean, Black Sea), the Okubo-Weiss eddy detection tool fails to project them before running the eddy-detection algorithm	Tools - Data Products - Aviso		Defect	jjr8	new	2011-04-13T12:49:44-0400	2011-04-13T12:49:44-0400	"Although the tool appears to work for these datasets, there is a caveat. The tool is designed to operate on projected data. Internally, the tool is supposed to project the geographic data to a coordinate system such as Albers Equal Area before running the eddy detection step, but I did not have time to finish this capability before I had to release MGET 0.8a25. The result is that all the physical calculations are being performed as if the cells are 0.125 meters wide and high, when they should actually be about 10000 meters. That is a big discrepancy, so why does it seem to work?

The answer is that, by default, the algorithm identifies eddies using a relative threshold, not an absolute one. The default threshold is 0.2 standard deviations below the mean, which is a common threshold from the literature. So even though the calculations of vorticity, shear, the Okubo-Weiss parameter, and have the wrong order of magnitude, the threshold still works because it is a unit-less fraction relative to the mean. However, if you instructed the tool to use an absolute threshold and provided a proper value (such as the one used by Chelton et al. 2007), it would not work properly. The results would be totally wrong; either no eddies would be detected, or all possible eddies would be detected, I’m not sure which.

The possible solutions are:

 1. Implement the automatic projecting of the data that I described.

 2. Limit the tool so that if a geographic dataset is selected, it will only allow the ""Standard deviation"" threshold type to be used (""Absolute"" would not be allowed).

The second is certainly less work. I am inclined to choose that one once I review the code and convince myself that it is really ok for the algorithm operate on geographic coordinates rather than projected coordinates. (I think it should be fine, because the datasets cover such a small extent, so the cells are all approximately the same size in both geographic and projected systems...)"	jjr8
6	0.8 Release	475	Long-running child processes (such as Okubo-Weiss eddy detection) cannot be cancelled from ArcGIS	Tools - Data Management		Defect	jjr8	new	2011-04-13T13:04:09-0400	2011-04-13T13:04:09-0400	The problem is that the {{{GeoEco.DataManagement.Processes.ChildProcess}}} class does not check for cancellation in its event loop. It should be simple enough to have it check once a second. Once that is done, any MGET tool that creates a child process will be cancellable.	jjr8
6	0.8 Release	476	Okubo-Weiss eddy detection tool should report progress	Tools - Oceanographic - Eddies		Enhancement	jjr8	new	2011-04-13T13:10:02-0400	2011-04-13T13:10:02-0400	"Eddy detection is CPU intensive and can take many minutes if many SSH images must be processed. Currently, the tool simply reports ""Detecting eddies"" and that's it, until the eddy detection step is complete."	jjr8
6	0.8 Release	477	"Okubo-Weiss eddy detection tool fails with ""python.exe: RuntimeError: ??? Index exceeds matrix dimensions"""	Tools - Oceanographic - Eddies		Defect	jjr8	new	2011-04-13T14:15:18-0400	2011-04-13T14:15:18-0400	"Here is full output:

{{{
Executing (Find Okubo-Weiss Eddies in Aviso SSH Product): AvisoGriddedSSHFindOkuboWeissEddies aviso-users grid2010 ""Mediterranean DT-Upd Merged MSLA SSH"" C:\Temp\Aviso2 Replace %(Region)s;%(DataSeries)s;%(Satellites)s;%(Variable)s_%(ImageType)s;%%Y;%(DataSeries)s_%(Satellites)s_%(Variable)s_%(ImageType)s_%%Y%%j.img ""Standard Deviation"" -0.2 # # # # # Degrees 1/1/2009 12/31/2009 60 120 C:\Temp\Aviso2\OPeNDAP_Cache true true false false false C:\Temp\Aviso2
Start Time: Wed Apr 13 13:47:12 2011
Running script AvisoGriddedSSHFindOkuboWeissEddies...
Querying time slices of Okubo-Weiss eddies in Aviso Mediterranean DT-Upd Merged MSLA SSH, clipped to indices tMin = 847, tMax = 898.
Query complete: 0:00:00 elapsed, 52 datasets found, 0:00:00.001903 per dataset.
Importing 52 datasets into ArcGIS Folder C:\Temp\Aviso2 with mode ""replace"".
Reading 52 time slices of Aviso Mediterranean DT-Upd Merged MSLA SSH, clipped to indices tMin = 847, tMax = 898.
Finished reading: 0:00:00 elapsed, 52 time slices read, 0:00:00.003942 per time slice.
Detecting eddies.
python.exe: RuntimeError: ??? Index exceeds matrix dimensions.
python.exe: 
python.exe: Error in ==> IdentifyOkuboWeissEddies at 195
RuntimeError: python.exe returned exit code 1, indicating failure.
The following consequences resulted from the original error:
Execution of C:\Python25\python.exe failed.
Import stopped before all datasets were imported: 0:00:29 elapsed, 0 datasets imported, 0:00:00 per dataset, 52 datasets not imported.
<type 'exceptions.RuntimeError'>: python.exe returned exit code 1, indicating failure.
Failed to execute (Find Okubo-Weiss Eddies in Aviso SSH Product).
End Time: Wed Apr 13 13:47:43 2011 (Elapsed Time: 31.00 seconds)
}}}

This comes from the core algorithm implemented in MATLAB (line 195 of the code) and has something to do with how eddies are tracked between successive SSH images. I do not know what the problem is at this time, but it appears to happen when you use the default parameters for the Okubo-Weiss tool. Those parameters do not specify a limit on the minimum sizes of eddies, so eddies as small as 1 pixel are permitted. If you provide a larger value for Minimum Area of Eddy Cores (I recommend at least 4), the error does not seem to happen."	jjr8
6	0.8 Release	514	Aviso data should be georeferenced using the TOPEX/Poseidon ellipsoid, not the WGS 1984 ellipsoid	Tools - Data Products - Aviso		Defect	jjr8	new	2012-01-05T10:58:14-0500	2012-01-05T10:59:33-0500	"A few years ago I tried to find out from Aviso what reference ellipsoid was used for the Mercator-projected merged satellite products. I could not obtain a satisfactory answer--I could not figure out how to get the person I was communicating with to understand the question. But recently I inquired again, after getting a helpful answer about the problem described in #511.

On 5 January 2012, I received email from Francoise Mertz at Aviso
User Services saying ""In the DUACS products (SLA, MSLA; Corssh are
not concerned) , all the missions (ERS-1, ERS-2, Envisat,GFO) are
adjusted on the ellipsoid of the reference mission (TP,J1 and J2)
which is: revolution with equatorial radius of 6378.1363 kilometers
and a flattening coefficient of 1/298.257.""

Those are the parameters for the standard TOPEX/Poseidon ellipsoid,
as can be seen in Table 6 of Tapley et al. (1994).

Tapley, B.D, Ries, J.C., Davis, G.W., Eanes, R.J., Schutz, B.E.,
Shum, C.K., Watkins, M.M., Marshall, J.A., Nerem, R.S., Putney,
B.H., Klosko, S.M., Luthcke, S.B., Pavlis, D.E., Williamson, R.G.
and Zelensky, N.P. (1994) Precision Orbit Determination for
TOPEX/POSEIDON. Journal of Geophysical Research 99: 24383-24404.

In Aviso's netCDF files, the lower-left corner of their global grid
is longitude 0, latitude -82. Note that until January 2012, Aviso's
FAQ stated:

    Question: In the gridded data, to which point lat/lon refer to?[[BR]]
    Answer: The lon/lat given is the SW corner of the box, as
    specified in the metadata of the NetCDF file

I suspected that was wrong and Francoise confirmed my suspicion,
saying ""indeed, you are right: in the gridded data, the data are
stored as points and not boxes that means that the lat/lon refers to
the exact point of the given value (and not the SW corner as
mentionned in the FAQ)."" (This issue may be discussed more in #511.)

So, given that the !CornerCoords property stores the x and y
coordinates of the center of the lower-left cell, we simply need to
transform lon 0, lat -82 into the Mercator system:

{{{
#!python
>>> import math
>>> from GeoEco.Datasets import Dataset
>>> sr1 = Dataset.ConvertSpatialReference('proj4', '+proj=latlong +a=6378136.3 +rf=298.257 +no_defs', 'obj')              # If you use WGS 1984 instead (i.e. replace ""+a=6378136.3 +rf=298.257"" with ""+ellps=WGS84 +datum=WGS84"", the results are the same
>>> sr2 = Dataset.ConvertSpatialReference('proj4', '+proj=merc +a=6378136.3 +rf=298.257 +lon_0=180.0 +no_defs', 'obj')
>>> transformer = Dataset._osr().CoordinateTransformation(sr1, sr2)
>>> xLLC, yLLC = transformer.TransformPoint(0.0, -82.0)[:2]
>>> xLLC, yLLC
(-20037506.143674389, -16925420.02271352)
}}}

We can also compute the x and y !CoordIncrements:

{{{
#!python
>>> 6378136.3 * 2 * math.pi / 360 / 3                   # 1/3 degree cell size at equator (1080 columns)
37106.492858656267
}}}

MGET's Aviso code should be corrected to use the TOPEX/Poseidon ellipsoid ({{{sr2}}} in the code above) and the !CornerCoords and !CoordIncrements I computed above."	jjr8
6	0.8 Release	518	MGET tools should utilize the ArcGIS progress bar (ArcGIS 9.3 and higher)	Core - ArcGIS Interop		Enhancement	jjr8	assigned	2012-02-16T11:37:05-0500	2012-02-20T16:05:16-0500	MGET already has an internal progress reporting system. We should use it to control the ArcGIS progress bar using the geoprocessor functions introduced in ArcGIS 9.3 (!SetProgressor, !SetProgressorPosition, !ResetProgressor).	jjr8
6	0.8 Release	523	Okubo-Weiss eddy detection tools should allow output of the Okubo-Weiss parameter	Tools - Oceanographic - Eddies		Enhancement	jjr8	new	2012-02-16T18:42:10-0500	2012-03-28T13:45:12-0400	This is useful if you want to detect rings around the eddy cores, as Ango Hsu did in his paper.	jjr8
6	0.8 Release	532	Create Lines From Vector Component Rasters tool should support non-square cells	Tools - Spatial Analysis		Enhancement	jjr8	assigned	2012-03-26T12:19:46-0400	2012-03-28T13:44:06-0400	Certain netCDF datasets (e.g. attached HF radar) do not have square cells. Currently, the tool assumes the cells are square; when they are not, it geolocates the vectors at the wrong locations. The tool should be changed to work properly with non-square cells.	jjr8
6	0.8 Release	536	When predictor rasters are in a geodatabase, Predict GLM / Predict GAM / Predict Tree fail with RPy_RException: Error in .local(.Object, ...) :    `C:\...' does not exist in the file system, and is not recognised as a supported dataset name.	Tools - Statistics		Defect	jjr8	new	2012-03-28T13:31:11-0400	2012-03-28T13:31:11-0400	The problem is that the {{{rgdal}}} R package cannot read rasters from geodatabases. MGET should temporarily copy the predictor rasters out of geodatabases before calling the R code for the prediction.	jjr8
6	0.8 Release	546	Tools that create R plots should use anti-aliasing	Tools - Statistics		Enhancement	jjr8	new	2012-04-29T17:04:17-0400	2012-04-29T17:04:17-0400	The default R {{{png}}} device does not implement anti-aliasing. My solution to making high quality R figures in PNG format has been to increase the DPI to 1000. But the R {{{cairoDevice}}} package can apparently create anti-aliased PNGs. Using this would make it feasible to reduce the default DPI and still produce nice figures.	jjr8
6	0.8 Release	549	Use geographiclib for accurate geodesic distance and area calcuations	Tools - Spatial Analysis		Task	jjr8	new	2012-07-06T12:12:40-0400	2012-07-27T15:43:53-0400	"Recently bumped into an excellent Python module to very accurately calculate distance and area from geographic points: [http://geographiclib.sourceforge.net/html/other.html#python geographiclib]. Unlike the distance functions in ArcGIS like Near, this library calculates shortest geodesic distance between any two points on the globe without suffering issues around the dateline. Attached are images showing the difference, both calculating nearest neighbor distance from the star point in the lower left. I include the toolbox models and Python script for generating both in the zip.

{{{
#!python
# import library
from geographiclib.geodesic import Geodesic

# The geodesic inverse problem
Geodesic.WGS84.Inverse(-41.32, 174.81, 40.96, -5.50)
# The geodesic direct problem
Geodesic.WGS84.Direct(40.6, -73.8, 45, 10000e3)

# How to obtain several points along a geodesic
line = Geodesic.WGS84.Line(40.6, -73.8, 45)
line.Position( 5000e3)
line.Position(10000e3)

# Computing the area of a geodesic polygon
def p(lat,lon): return {'lat': lat, 'lon': lon}
Geodesic.WGS84.Area([p(0, 0), p(0, 90), p(90, 0)])
}}}

Note that interpolating along a line is done with the direct geodesic function from a starting point and by azimuth and distance, which you can get between two points using the inverse geodesic function. I do this in another script with a list comprehension on a [http://pandas.pydata.org/pandas-docs/stable/dsintro.html#dataframe PANDAS data frame]:

{{{
#!python
d['km'] = [geod.Inverse(r['lat1'], r['lon1'], r['lat2'], r['lon2'])['s12'] / 1000 for i,r in d.iterrows()]
}}}

I also like that these techniques are supported by very recent peer reviewed publications:

 * C. F. F. Karney, [http://arxiv.org/abs/1102.1215v1 Geodesics on an ellipsoid of revolution], Feb. 2011; resource page [http://geographiclib.sf.net/geod.html geod.html].

 * C. F. F. Karney (2012) [http://dx.doi.org/10.1007/s00190-012-0578-z Algorithms for geodesics]. Journal of Geodesy.


'''ArcGIS Near'''

[[Image(ptdist_ArcGIS_Near.png,800px)]]

'''VS.'''

'''Python GeographicLib::Geodesic'''

[[Image(ptdist_Python_Geodetic.png,800px)]]"	bbest
6	0.8 Release	555	Add support for numpy 1.7.0	Core - Other		Task	jjr8	new	2013-03-14T07:24:53-0400	2013-03-14T07:24:53-0400		jjr8
6	0.8 Release	557	Predict XXXXX From Table tools fail with RPy_RException: Error in prediction(response$predicted, response$actual) : Number of classes is not equal to 2.	Unknown		Defect	jjr8	new	2013-03-15T18:31:31-0400	2013-03-15T18:31:31-0400	"Example output from ArcGIS:

{{{
Executing (Predict Random Forest From Table (2)): RandomForestModelPredictFromArcGISTable C:\Temp\Sajad\Model\rf1.rdata C:\Temp\Sajad\Geodatabase.mdb\Test # # # true # # tpr fpr true # 1000 3000 3000 10 white
Start Time: Thu Mar 14 22:14:45 2013
Running script RandomForestModelPredictFromArcGISTable...
Loaded randomForest.formula from C:\Temp\Sajad\Model\rf1.rdata. The model was fitted with the R randomForest package.
Querying ArcGIS table ""C:\Temp\Sajad\Geodatabase.mdb\Test"".
Finished retrieving rows: 0:00:10 elapsed, 4630 rows retrieved, 0:00:00.002234 per row.
Predicting...
RPy_RException: Error in prediction(response$predicted, response$actual) : 
  Number of classes is not equal to 2.
ROCR currently supports only evaluation of binary classification tasks.
Obtaining additional debugging information. This may cause this tool to hang. If you do not see any additional messages after one minute, halt its execution.
<class 'rpy.RPy_RException'>: Error in prediction(response$predicted, response$actual) : 
  Number of classes is not equal to 2.
ROCR currently supports only evaluation of binary classification tasks.

Failed to execute (Predict Random Forest From Table (2)).
Failed at Thu Mar 14 22:14:58 2013 (Elapsed Time: 13.00 seconds)
}}}

The error is either a bug in MGET or a problem with what you are doing, depending what you provide for the “Field To Receive The Predicted Values Of The Response Variable” parameter.

If you leave this parameter blank, then MGET expects that there will be a field in the table for the response variable and that it will contain the actual values (a.k.a. observed values) of the test data. It then predicts the test data from the other fields, compares the predicted values to the actual values, and produces summary statistics for that comparison. (This is the same thing that happens if you leave the Input Table parameter empty, but in that case, the prediction and comparison is done with the training data which was stored in the input model file by the Fit Random Forest tool.)

If you do not leave the parameter blank, then MGET is supposed to perform the predictions and store the values in that field. Since no actual values are available, it does not print a statistical summary. The purpose of this is for you to obtain the values and perform your own statistical summary of the prediction (you could calculate the residuals, for example) or for you to predict results for data that is not test or training, when you have deployed the model into a resource management application, for example.

Currently, under both cases, MGET may report the same error. Why?

In the first case (you leave the parameter blank), if you get the error it indicates that the field for the response variable contains NULL values for all records. There are no actual values on which the prediction can be performed. MGET fails to detect this situation and asks ROCR to compute summary statistics. ROCR fails because there are no records that have values.

In the second case (you give a field to populate with predicted values), it is clearly a bug in MGET. Because the field to receive the values has the same name as the field that was used to fit the model, MGET gets confused and handles the scenario as if you left the parameter blank, leading to the same situation as above.
"	jjr8
7	0.8 Release	524	Run Coral Reef Connectivity Simulation reports Warning: Size vector should be a row vector with integer elements.	Tools - Connectivity - Coral Reefs		Defect	jjr8	assigned	2012-02-16T18:44:10-0500	2012-02-20T16:15:23-0500		jjr8
6	Unscheduled Release	4	setup.py should generate list of optional software in InstallationGuide.html	Documentation		Task	jjr8	assigned	2006-12-07T17:47:01-0500	2007-09-18T11:06:34-0400	setup.py should generate list of optional software in !InstallationGuide.html. This list should be assembled by querying the Dependency metadata for all !GeoEco methods.	jjr8
6	Unscheduled Release	9	Add metadata for GeoEco base modules	Core - Metadata & Types		Task	jjr8	assigned	2006-12-07T18:30:58-0500	2007-09-18T11:13:34-0400	Adding metadata to the base modules allows them to be documented automatically by the build script.	jjr8
6	Unscheduled Release	21	Add support for more reStructuredText primitives for documentation stored in ArcGIS toolbox	Documentation		Task	jjr8	assigned	2006-12-11T11:06:58-0500	2007-09-18T11:12:26-0400	The XSL transform in CreateArcGISToolbox should be fleshed out to support as many reStructuredText primitives as practical	jjr8
6	Unscheduled Release	32	GeoEco type library version number should be automatically assigned	Core - COM Interop		Task	jjr8	assigned	2007-01-08T13:34:05-0500	2007-09-18T11:15:38-0400	Currently, the !GeoEco type library version number is hard coded in COM.py. It should be automatically assigned by the build system. It is unclear what it should be assigned to--maybe the major and minor version numbers of !GeoEco itself?	jjr8
6	Unscheduled Release	33	Python exceptions should be translated into COMExceptions when a class is invoked through COM	Core - COM Interop		Task	jjr8	assigned	2007-01-08T17:01:18-0500	2007-09-18T11:15:58-0400	!GeoEco.COM.!ExceptionTranslatingDispatcher or !MetadataDesignatedWrapPolicy need to translate Python exceptions into win32com.server.COMExceptions.	jjr8
6	Unscheduled Release	35	VBScript can't assign individual array elements of list property exposed through COM	Core - COM Interop	0.2	Defect	jjr8	assigned	2007-01-09T16:26:17-0500	2007-09-18T11:16:36-0400	"If a COM-exposed class exposes a read/write property that is a list of integers, VBScript can't assign individual array elements, like this:

{{{
v1 = obj.IntegerListProperty
obj.IntegerListProperty(0) = 10
}}}

This is supposed to assign element 0 to the value 10. In this case, VBScript calls !InvokeEx like this:

   2007-01-09 16:14:14,186 DEBUG COM object 0x01A3B910: !InvokeEx called: dispid = 3 (!IntegerListProperty), lcid = 1033, wFlags = 4 (DISPATCH_PROPERTYPUT), args = '''''(0, 10)''''', kwArgs = None, serviceProvider = <PyIServiceProvider at 0x1a247b4 with obj at 0x3dfc8>

VBScript passes in two parameters, the index and the value. Pythoncom passes this as a tuple to the fset method, like this: {{{args = (0, 10)}}}. The fset method thinks the caller is assigning a new array with two elements, rather than trying to set the value of an element in the existing array.

I need to figure out how to expose array properties properly."	jjr8
6	Unscheduled Release	38	C# calls to COM-exposed methods that return multiple results ([in, out] in IDL) raise E_UNEXPECTED	Core - COM Interop	0.2	Defect	jjr8	assigned	2007-01-22T10:41:08-0500	2007-09-18T11:09:33-0400	"When a COM-exposed method returns one result, it is encoded as {{{[out, retval]}}} in IDL, and is the return value of the method. Additional results are encoded as {{{[in, out]}}} method parameters. The !GeoEco COM infrastructure does not pass the input values to the method, effectively making them {{{[out]}}} parameters.

As of [36], this all works fine for COM clients that call through the COM {{{IDispatch}}} interface. But when C# clients place a ""vtable"" call, E_UNEXPECTED is raised and the call fails. This kind of call is handled by pythoncom's ""universal dispatcher"", which is different than the code that handles the {{{IDispatch}}} calls. I reviewed its Python and C++ code and could not isolate the failure. I will have to contact Mark Hammond with a repro scenario and hopefully he'll fix it.

We need a plan in the interim. It is quite likely that we'll have methods that want to return multiple output parameters. I will have to think about this..."	jjr8
6	Unscheduled Release	39	setup.py should include client-side C++ header files in COM directory, not just the IDL and TLB	Core - COM Interop		Task	jjr8	assigned	2007-01-28T11:32:24-0500	2007-09-18T11:16:50-0400	Currently, {{{setup.py}}} generates {{{GeoEco.idl}}} and {{{GeoEco.tlb}}} in the COM directory. This is fine for early-bound clients that can read type libraries, such as C# and VB.Net. But C++ clients would prefer to include a header file that defines the GeoEco interfaces. This header file is generated by the midl.exe compiler but {{{setup.py}}} currently deletes it. Instead, it should include it in the COM directory.	jjr8
6	Unscheduled Release	46	CreateArcGISToolbox.exe should read MinValue and MaxValue and create GPRangeDomainClass	Build System		Task	jjr8	assigned	2007-02-09T14:28:09-0500	2007-09-18T11:18:38-0400	"The {{{PythonFloat}}} and {{{PythonInteger}}} classes in {{{GeoEco.Types}}} have {{{MinValue}}} and {{{MaxValue}}} properties that are checked in {{{ValidateValue}}}. {{{CreateArcGISToolbox.exe}}} should deserialize these properties from {{{Metadata.xml}}} and create {{{GPRangeDomainClass}}} instances on the tool parameters.

The main issue here is that {{{MinValue}}} and {{{MaxValue}}} can be None. This prevents them from being XML-serialized as a normal {{{double}}}. I need to figure out how to deal with this..."	jjr8
6	Unscheduled Release	49	All classes should use Logger.RaiseException rather than calling Python's raise statement directly	Core - Other		Task	jjr8	assigned	2007-02-27T16:35:29-0500	2007-09-18T11:18:50-0400	All classes should use {{{Logger.RaiseException}}} rather than calling Python's {{{raise}}} statement directly, so that proper logging occurs. I thought I had made this change, but I discovered some cases of direct uses of {{{raise}}} in {{{ArcGIS.py}}}. I need to check the entire source tree again.	jjr8
6	Unscheduled Release	53	Investigate Python 2.5 enhancements to distutils metadata	Setup & Installation		Task	jjr8	assigned	2007-03-15T10:52:12-0400	2007-09-18T11:24:37-0400	"We should investigate the following enhancement to Python 2.5 {{{distutils}}}:

""Some simple dependency support was added to Distutils. The setup() function now has requires, provides, and obsoletes keyword parameters. When you build a source distribution using the sdist command, the dependency information will be recorded in the PKG-INFO file. 

Another new keyword parameter is download_url, which should be set to a URL for the package's source code. This means it's now possible to look up an entry in the package index, determine the dependencies for a package, and download the required packages. 

{{{
#!python
VERSION = '1.0'
setup(name='PyPackage', 
      version=VERSION,
      requires=['numarray', 'zlib (>=1.1.4)'],
      obsoletes=['OldPackage']
      download_url=('http://www.example.com/pypackage/dist/pkg-%s.tar.gz'
                    % VERSION),
     )
}}}

Another new enhancement to the Python package index at http://cheeseshop.python.org is storing source and binary archives for a package. The new upload Distutils command will upload a package to the repository. 

Before a package can be uploaded, you must be able to build a distribution using the sdist Distutils command. Once that works, you can run python setup.py upload to add your package to the PyPI archive. Optionally you can GPG-sign the package by supplying the --sign and --identity options. """	jjr8
6	Unscheduled Release	54	Take advantage of Windows-specific options of distutils	Setup & Installation		Task	jjr8	assigned	2007-03-16T16:43:33-0400	2007-09-18T11:24:46-0400	"Python's {{{distutils}}} has various Windows-specific options that make the Windows Installer package more user-friendly, including:

 * {{{--bitmap}}} - By default the installer will display the cool ``Python Powered'' logo when it is run, but you can also supply your own bitmap which must be a Windows .bmp file with the --bitmap option. 

 * {{{--title}}} - The installer will also display a large title on the desktop background window when it is run, which is constructed from the name of your distribution and the version number. This can be changed to another text by using the --title option."	jjr8
6	Unscheduled Release	58	Contact Information should be added to metadata	Core - Metadata & Types		Task	jjr8	assigned	2007-03-27T10:27:26-0400	2007-09-18T11:27:05-0400	The metadata for classes, properties and methods should include contact information, so the user knows who to email when problems occur.	jjr8
6	Unscheduled Release	59	HDF to ArcGIS Raster tool should store HDF attributes in raster's metadata.xml file	Tools - Conversion		Task	jjr8	assigned	2007-04-05T16:36:21-0400	2007-09-18T11:22:08-0400	"In the old tool package, Marine Ecology Tools, the HDF to Raster tool extracted the HDF attributes and stored them in the raster's metadata.xml file. I disabled this code after I found out that ArcGIS's built-in Copy Raster tool failed to copy the metadata in metadata.xml when it copied the raster. (What an annoying bug).

I should re-enable the HDF attribute extraction code in the new MGET HDF to ArcGIS Raster tool. I will open a separate ticket for enhancing MGET's own Copy Raster tool, saying that the tool should manually copy the metadata, to work around the ArcGIS bug."	jjr8
6	Unscheduled Release	60	ArcGISRaster.Copy should manually copy metadata in raster's metadata.xml	Tools - Data Management		Task	jjr8	assigned	2007-04-05T16:39:23-0400	2007-09-18T11:22:24-0400	As described in ticket #59, the HDF to ArcGIS Raster tool will store HDF attributes as metadata in the output raster's metadata.xml file. But the ArcGIS built-in Copy Raster tool fails to copy the metadata - it only copies the raster. The MGET ArcGISRaster.Copy method should manually copy the metadata, to work around this ArcGIS bug.	jjr8
6	Unscheduled Release	62	Build system should generate COM Reference documentation	Documentation		Task	jjr8	assigned	2007-05-07T11:29:26-0400	2007-09-18T11:28:23-0400	The build system should generate COM Reference documentation from the !GeoEco class metadata, just like is done for ArcGIS and Python Reference documentation.	jjr8
6	Unscheduled Release	68	Cannot set or get Geoprocessor property of GeoEco.GeoprocessorManager COM object	Core - COM Interop	0.2	Defect	jjr8	assigned	2007-05-31T09:30:00-0400	2007-09-18T11:07:44-0400	"This VB code:

{{{
#!vb
Set gp = CreateObject(""esriGeoprocessing.GPDispatch.1"")
Set gpm = CreateObject(""GeoEco.GeoprocessorManager"")
gpm.Geoprocessor = gp
}}}

produces this error:

{{{
C:\temp6>cscript test2.vbs
Microsoft (R) Windows Script Host Version 5.6
Copyright (C) Microsoft Corporation 1996-2001. All rights reserved.

Traceback (most recent call last):
  File ""C:\Python25\Lib\site-packages\win32com\server\dispatcher.py"", line 77, in _InvokeEx_
    return self.policy._InvokeEx_(dispid, lcid, wFlags, args, kwargs, serviceProvider)
  File ""C:\Python25\Lib\site-packages\win32com\server\policy.py"", line 332, in _InvokeEx_
    return self._invokeex_(dispid, lcid, wFlags, args, kwargs, serviceProvider)
  File ""C:\Python25\Lib\site-packages\GeoEco\COM.py"", line 271, in _invokeex_
    returnedValue = win32com.server.policy.DesignatedWrapPolicy._invokeex_(self, dispid, lcid, wFlags, args, kwArgs, serviceProvider)
  File ""C:\Python25\Lib\site-packages\win32com\server\policy.py"", line 618, in _invokeex_
    if type(getattr(self._obj_, name, None)) == types.MethodType and \
TypeError: 'classmethod' object is not callable
pythoncom error: Python error invoking COM method.

Traceback (most recent call last):
  File ""C:\Python25\Lib\site-packages\win32com\server\dispatcher.py"", line 79, in _InvokeEx_
    return self._HandleException_()
  File ""C:\Python25\Lib\site-packages\win32com\server\dispatcher.py"", line 129, in _HandleException_
    reraise()
  File ""C:\Python25\Lib\site-packages\win32com\server\dispatcher.py"", line 77, in _InvokeEx_
    return self.policy._InvokeEx_(dispid, lcid, wFlags, args, kwargs, serviceProvider)
  File ""C:\Python25\Lib\site-packages\win32com\server\policy.py"", line 332, in _InvokeEx_
    return self._invokeex_(dispid, lcid, wFlags, args, kwargs, serviceProvider)
  File ""C:\Python25\Lib\site-packages\GeoEco\COM.py"", line 271, in _invokeex_
    returnedValue = win32com.server.policy.DesignatedWrapPolicy._invokeex_(self, dispid, lcid, wFlags, args, kwArgs, serviceProvider)
  File ""C:\Python25\Lib\site-packages\win32com\server\policy.py"", line 618, in _invokeex_
    if type(getattr(self._obj_, name, None)) == types.MethodType and \
<type 'exceptions.TypeError'>: 'classmethod' object is not callable
C:\temp6\test2.vbs(3, 1) Python COM Server Internal Error: Unexpected Python Error: <type 'exceptions.TypeError'>: 'classmethod' object is not callable
}}}

I am not sure what is going on here and need to investigate."	jjr8
6	Unscheduled Release	69	DateTimeTypeMetadata should support default values, minValue and maxValue	Core - Metadata & Types		Task	jjr8	assigned	2007-05-31T15:47:46-0400	2007-09-18T11:30:05-0400	"Currently, !DateTimeTypeMetadata does not allow you to specify a default value for a method argument, nor does it propagate the minValue and maxValue into Metadata.xml, which would allow CreateArcGISToolbox to configure a domain.

I did not implement these features because they are not currently needed, and it is not clear how datetime values should be written in XML (it should probably be ISO 8601 format, but it is not clear how well this can be deserialized by the built-in .NET deserialization code)."	jjr8
6	Unscheduled Release	70	Re-enable DTD validation for XSL processing done by setup.py	Build System		Task	jjr8	assigned	2007-06-01T12:56:52-0400	2007-09-18T11:30:20-0400	"{{{setup.py}}} calls {{{xsltproc}}} for a variety of XSL processing. I had to disable DTD validation because the DTDs were being fetched over the network on every invocation of {{{xsltproc}}}. This was really slow, but also www.w3.org kept banning my IP address for 24 hours for ""abuse"" of their server.

I should investigate how to cache the DTDs locally. There are several ways, including command-line options to {{{xsltproc}}} and the XSL_CATALOG_FILES environment variable."	jjr8
6	Unscheduled Release	74	Add additional validation to BatchProcessing.GenerateForMethod	Core - Batch Processing		Task	jjr8	assigned	2007-06-15T14:50:38-0400	2007-09-18T11:30:39-0400	"{{{BatchProcessing.GenerateForMethod}}} contains a lot of validation but barely any related to the ""find and process"" method, which is the most complicated and important batch-processing method. I should complete the validation code, to prevent developers from introducing bugs that are difficult to diagnose."	jjr8
6	Unscheduled Release	77	Setting the geoprocessor's OverwriteOutput property from a Python script does not seem to work	Core - ArcGIS Interop	0.2	Defect	jjr8	assigned	2007-06-19T14:36:31-0400	2007-09-18T11:07:56-0400	"In the script below, if the files in inputFiles has already been copied to outputFiles, the script will fail with an error indicating that a destination file already exists. This is not supposed to happen because the geoprocessor's OverwriteOutput environment variable is supposed to be used by FileFindAndCopy_GeoEco to determine whether files should be overwritten.

{{{
#!python
from win32com.client import Dispatch

gp = Dispatch('esriGeoprocessing.GPDispatch')
gp.AddToolbox('c:\\Python25\\lib\\site-packages\\GeoEco\\ArcGISToolbox\\Marine 
print(str(gp.OverwriteOutput))
gp.OverwriteOutput = True
print(str(gp.OverwriteOutput))
gp.FileFindAndCopy_GeoEco('c:\\temp8\\inputFiles', 'c:\\temp8\\outputFiles')
}}}

The print statements show that the OverwriteOutput environment variable is being set. But enabling debug logging reveals that FileFindAndCopy_GeoEco apparently is not seeing the True value:

{{{
2007-06-19 14:28:41,415 DEBUG ArcGIS Geoprocessor object 0x00EB0E90: Get OverwriteOutput returned False
2007-06-19 14:28:41,415 DEBUG Parameter overwriteExisting = False
}}}

At first glance, this looks like a bug in Arc 9.1."	jjr8
6	Unscheduled Release	78	"The ""Scripting Syntax"" section of the ArcGIS documentation should say ""TRUE"" should be passed for boolean parameters"	Core - ArcGIS Interop		Task	jjr8	assigned	2007-06-19T14:51:43-0400	2007-09-18T11:38:44-0400	"Apparently, when you call a script tool through the ArcGIS geoprocessor's COM Automation object, you cannot pass a VARIANT_BOOL to indicate True or False. Instead you must pass the case-insensitive string ""TRUE"" to indicate True or anything else to indicate False (including ""FALSE"", or even VARIANT_BOOL of True!)

I need to investigate further to determine if this only happens with Python (indicating an issue with pythoncom) or with other scripting languages (indicating that this is how the geoprocessor COM Automation object really works).

If this is really how the geoprocessor COM Automation object really works, the ArcGIS documentation should note this issue under the ""Scripting Syntax"" section, for all boolean parameters of all tools. The XSL should automatically insert a statement describing the situation."	jjr8
6	Unscheduled Release	83	CreateArcGISToolbox.exe should be built with .NET Framework 1.1, not 2.0	Build System		Task	jjr8	assigned	2007-07-09T17:41:14-0400	2007-09-18T11:32:09-0400	Currently, CreateArcGISToolbox.exe is being compiled with Visual Studio 2005, which forces the project to reference the .NET Framework 2.0. But to preserve ArcGIS 9.1 compatibility, I am building MGET on a machine that has ArcGIS 9.1 installed. ArcGIS 9.1 only supports .NET Framework 1.1. Thus, CreateArcGISToolbox.exe is currently violating ESRI's compatibility support statement by referencing .NET 2.0. It should be recompiled against .NET 1.1.	jjr8
6	Unscheduled Release	85	Review all calls to os.stat and ensure they can handle the returned 64-bit integers	Core - Other		Task	jjr8	assigned	2007-08-01T12:52:34-0400	2007-09-18T11:32:29-0400	The Python os.stat method returns a 64-bit integer for the file size. This is good because occasionally GIS data files get very large -- greater than the 2 GB limit that would be imposed if file sizes were restricted to the range of 32-bit signed integers. I need to review all GeoEco code to ensure calls to os.stat are expecting the 64-bit integer.	jjr8
6	Unscheduled Release	86	Batched functions of the File class are missing the refreshArcGISCatalog parameter	Tools - Data Management	0.2	Defect	jjr8	assigned	2007-08-09T10:53:59-0400	2007-09-18T11:08:22-0400	"The non-batched functions have the refreshArcGISCatalog parameter, for example:

{{{
Copy(sourceFile, destinationFile[, overwriteExisting[, refreshArcGISCatalog]]) 

}}}

But this parameter is missing from the batched functions:

{{{
CopyList(sourceFileList, destinationFileList[, skipExisting[, overwriteExisting]]) 

CopyTable(connection, table, sourceFileField, destinationFileField[, where[, orderBy[, directions[, skipExisting[, overwriteExisting]]]]]) 

CopyArcGISTable(table, sourceFileField, destinationFileField[, where[, orderBy[, directions[, skipExisting[, overwriteExisting]]]]]) 

}}}"	jjr8
6	Unscheduled Release	137	"""Find"" geoprocessing tools (e.g. ""Find Files"") should have a single output table parameter, not two parameters, output workspace and table name"	Tools - Data Management	0.3	Defect	jjr8	assigned	2007-10-31T16:10:54-0400	2009-05-28T12:11:34-0400	"When I created the Find geoprocessing tools, I copied the design pattern of the ArcGIS Create Table tool. That tool has two parameters: Output Location, the workspace in which the table should be created, and Output Table, the name of the table to create. The ArcGIS Create Feature Class tool is similar.

After a while I discovered that the other ArcGIS geoprocessing tools did not follow this design pattern. Instead, they simply had an Output Table parameter, data type Table, which was the path to the table to create. After working with this for months, I see no major advantage to the first design pattern, and the second is clearly simpler for the user to use and for me to develop.

The only place in MGET where I follow the first design pattern is the various ""Find"" functions:

 * Find !CoastWatch Files
 * Find !CoastWatch Images In Files
 * Find Directories
 * Find Files
 * Find Rasters

I should change these to the second design pattern. This may be complicated because the batch processing infrastructure relies on the Find functions (but I don't think it relies on the ArcGIS geoprocessing versions of the Find functions)."	jjr8
6	Unscheduled Release	147	"Sample Rasters Listed in Fields tool should have a ""skip missing rasters"" option"	Tools - Spatial Analysis	0.4a2	Enhancement	jjr8	assigned	2007-11-30T11:18:11-0500	2009-05-28T12:11:47-0400	"Currently, the Sample Rasters Listed in Fields tool will fail if any of the rasters do not exist. It just passes the list of rasters to the ArcGIS Spatial Analyst Sample tool and allows that tool to fail with an obscure error message. It would be better if Sample Rasters Listed in Fields tested the existence of each raster and had a ""skip missing rasters"" option.

This is feature would be useful when sampling time series rasters. Each point has a date and you might want to compute the raster to sample with the ArcGIS Calculate Fields tool. But you may not have rasters for all of the dates. It is a pain to then mark all of the points for which the raster does not exist (although this can be done if you use the Find Rasters tool and join it to the points table with Keep All = FALSE)."	jjr8
6	Unscheduled Release	172	Modify setup.py to automatically enumerate packages in the source tree	Setup & Installation	0.4	Enhancement	jjr8	assigned	2008-02-08T15:28:21-0500	2008-02-08T15:28:34-0500	Currently, when you add a new package to the !GeoEco source tree, you also have to modify the list of packages in setup.py. setup.py should be changed to automatically enumerate the packages in the source tree. This would eliminate the need for the manual step.	jjr8
6	Unscheduled Release	180	Calls to Logger.LogExceptionAsError should usually not report additional error messages	Core - Other	0.4	Defect	jjr8	assigned	2008-02-19T13:24:51-0500	2009-05-28T12:11:53-0400	"Many !GeoEco methods wrap their main processing in a try/except block that calls Logger.!LogExceptionAsError in the except handler. This is good. But they usually call that method with an additional log message. This causes the tool to report an ""The following consequences resulted from the original error:"" followed by the additional error message. As the stack unwinds, higher level methods do the same thing in their own except handlers, resulting in additional errors.

For example:

{{{

Extracting SDS ""asdf"" from HDF file C:\temp7\test.bin to binary file C:\temp7\test2.bin...
hdp.exe: 
hdp.exe: HDP ERROR>>> in dsd: C:\temp7\test.bin is not an HDF file.
hdp.exe: 
hdp.exe: HDP ERROR>>> in do_dumpsds: dsd failed.
RuntimeError: hdp.exe returned exit code 1, indicating failure.
The following consequences resulted from the original error:
Execution of C:\Python25\lib\site-packages\GeoEco\Bin\win32\hdp.exe failed.
Could not extract SDS ""asdf"" from HDF file C:\temp7\test.bin to binary file C:\temp7\test2.bin

}}}

The original design was intended to provide as much information as possible. I imagined scenarios where something failed very deep in the stack and the resulting error message had nothing to do with the complex operation being orchestrated further up in the stack. As the stack unwound, the additional error messages were supposed to show how the deep failure related to the higher level operations.

I now think this design does not provide much value. Most of the time, the low-level failure message does include enough information that a reasonably skilled user could figure out how to fix it. In this case, the additional error messages simply fill up the screen with intimidating text that distracts the user from the root problem. In the case of low-level messages that lack detail, the user often does not have much chance at resolving the problem anyway, unless they have a very high level of skill and turn on the debug logging.

The calls to Logger.!LogExceptionAsError should be reviewed, and most of them should be changed to not report additional error messages. In the example above, the output is compressed to:

{{{

Extracting SDS ""asdf"" from HDF file C:\temp7\test.bin to binary file C:\temp7\test2.bin...
hdp.exe: 
hdp.exe: HDP ERROR>>> in dsd: C:\temp7\test.bin is not an HDF file.
hdp.exe: 
hdp.exe: HDP ERROR>>> in do_dumpsds: dsd failed.
RuntimeError: hdp.exe returned exit code 1, indicating failure.

}}}"	jjr8
6	Unscheduled Release	181	"When a dependency fails for a specific parameter, the error message says ""this function requires ..."". It should say ""the X parameter requires...""."	Core - Metadata & Types	0.4	Defect	jjr8	new	2008-02-21T09:52:48-0500	2008-02-21T15:20:19-0500	The problem is that the current dependency-checking infrastructure outputs the same message in all circumstances. Because I originally did not have dependencies for individual parameters, all the messages are written as if it is only functions that can have dependencies. This should be fixed when I rework the dependency infrastructure.	jjr8
6	Unscheduled Release	186	Add GSHHS tool for converting river and shoreline databases	Tools - Data Products - GSHHS		Task	jjr8	assigned	2008-02-29T08:42:38-0500	2008-03-24T18:07:14-0400	The GSHHS 1.6 database contains databases of rivers and shorelines. I should add a tool to convert these.	jjr8
6	Unscheduled Release	219	Implement the GSHHS 1.4 Database to ArcGIS Polylines tool	Tools - Data Products - GSHHS		Enhancement	jjr8	new	2008-03-24T18:05:28-0400	2008-03-24T18:05:28-0400	GSHHS 1.4 Database to ArcGIS Polygons tool is slow and cannot handle the full resolution GSHHS database due to ArcGIS running out of memory(see #133). I should implement a version of this tool that creates polylines rather than polygons. It will probably be faster and not suffer from the memory problem.	jjr8
6	Unscheduled Release	223	Single Output Map Algebra tools should work around ArcGIS 9.1 bug where projection becomes <Undefined>	Tools - Spatial Analysis	0.4	Defect	jjr8	assigned	2008-03-25T09:08:07-0400	2009-05-28T12:12:02-0400	"As described in #156, the ArcGIS 9.1 Spatial Analyst Single Output Map Algebra tool sometimes causes the projection of the output raster to be <Undefined>. I don't know why this happens and I believe it is a bug in ArcGIS. In #156, I updated some of the conversion tools to work around this problem. But I never updated MGET's batched versions of Single Output Map Algebra, including:

 * Find Rasters and Execute Single Output Map Algebra
 * Single Output Map Algebra For Rasters Listed in Table
 * Single Output Map Algebra for Table Rows"	jjr8
6	Unscheduled Release	231	Tool for Projecting to Optimal Albers	Tools - Data Management		Task	jjr8	assigned	2008-04-08T12:16:48-0400	2009-05-28T12:12:36-0400	"This tool defines an optimal Albers projection in meters for minimizing linear/area distortion, based on the extent of the input geodataset.  So far, the idea is to use it with either a vector featureclass or raster dataset.  Output cell size is an additional parameter in the case of a raster dataset.  It uses the 1/6th rule for defining the latitudes, like so:

{{{
x1, y1, x2, y2 = [float(v) for v in desc.Extent.split()]
d['x_0'] = x1 + (x2 - x1)/2 # Central_Meridian
d['y_0'] = y1 + (y2 - y1)/2 # Latitude_Of_Origin
dy = (y2-y1) / 6
d['y_1'] = y1 + dy          # Standard_Parallel_1
d['y_2'] = y2 - dy          # Standard_Parallel_2
}}}

The 1st and 2nd parallels are defined as the min (y1) and max (y2) latitudes of the input extent + and - 1/6th of the max-min range (dy).  The same datum as the input is used so as to not require a transformation method.

This is all setup for the final project command:

{{{
# for raster
gp.ProjectRaster_management(in_geodataset, out_geodataset, out_prj, 'NEAREST', out_cellsize)

# for featureclass
gp.Project(in_geodataset, out_geodataset, out_prj)
}}}"	bbest
6	Unscheduled Release	234	Upgrade gzip DLL to 1.3.5	Core - Other		Task	jjr8	assigned	2008-04-23T10:45:42-0400	2009-05-28T12:12:45-0400	gzip 1.3.5 for Windows adds support for files > 4GB.	jjr8
6	Unscheduled Release	241	Sample Rasters tool should automatically create fields	Tools - Spatial Analysis	0.5a12	Enhancement	jjr8	assigned	2008-05-02T15:44:22-0400	2009-05-28T12:12:58-0400	"Currently, the Sample Rasters tool requires the user to specify the names of the fields to receive the sampled values (one field for each raster), and that these fields must already exist. The tool would be easier to use if the field names were optional entirely.

It would work like this:

 * If no field name was specified, the tool would use the name of the raster as the field name. The tool must be smart enough to coerce the raster name into a legal field name (e.g. shapefiles can only have 10 character field names; certain characters are illegal; the name must not duplicate a previously used name). The tool should issue an informational message to explain this was done.

 * If the field does not exist, the tool would create it automatically. It should examine the data type of the raster and create a field with the same data type. The tool should issue an informational message to explain this was done; ideally it should not duplicate the message above.

 * If the field does exist, the tool would compare its data type to that of the raster. If the raster data type can be coerced into the field data type with no loss of precision, the tool would do this automatically. If there was a loss of precision the tool would do it anyway, but issue a warning.

Thanks to Pat Iampietro for suggesting this enhancement."	jjr8
6	Unscheduled Release	253	Add cross validation option to Plot ROC and Plot Performance of Binary Classification Model tools	Tools - Statistics	0.5b1	Enhancement	jjr8	assigned	2008-05-21T10:07:35-0400	2009-05-28T12:13:22-0400	"Currently, these tools can operate on the data originally used to fit the model or a user-provided data set. Michelle indicated that a useful third option would be to operate on a ""drop-one"" cross validation of the original data. Here is some example code from her.

{{{
nr<- nrow(newdat)
predp_g1.4<- rep(NA,nr)

for(i in 1:nr)
{
    newdat_cv<- newdat[-i,]
    mod<- gam(OliveRidleyPresent~s(Backdown_ln,bs=""ts"")+s(setduration,bs=""ts"")+s(DTMSLAMinus15Dist,DTMSLAPlus15Dist,k=100,bs=""ts""),data=newdat_cv,family=binomial,gamma=1.4)
    predp_g1.4[i]<- predict.gam(mod,newdata=newdat[i,],type=""response"")
}

library(ROCR)
pred_final<- prediction(predp_g1.4,mod1$y)
perf1 <- performance(pred_final,""tpr"",""fpr"")
plot(perf1)
abline(0,1)

}}}"	jjr8
6	Unscheduled Release	254	Add optional weights and offset parameters to Fit GLM tool	Tools - Statistics	0.5b1	Enhancement	jjr8	assigned	2008-05-21T13:15:38-0400	2009-05-28T12:13:31-0400	Michelle indicated that the weights and offset parameters can be useful for fitting GLMs. For example, the offset parameter can be useful when fitting a binomial model when the response variable is not 0 or 1, but rather a count of 0 or 1 cases that came out as 1. In this experiment, each row of the input table would have not only the count of cases that came out as 1, but the total number of cases. This would be a separate field that should be passed in as the optional offset parameter. The weights parameter should also be an optional field.	jjr8
6	Unscheduled Release	256	Plot ROC tool should report the confidence intervals for the area under the curve (AUC)	Tools - Statistics	0.5b1	Enhancement	jjr8	assigned	2008-05-21T17:33:25-0400	2008-05-22T12:47:05-0400	"Michelle Sims indicated that there is a way to generate these intervals through cross validation or something. The idea is that if the CIs enclose the line y=x (AUC is 0.5) then the AUC of the model is not statistically significantly different from a model that produces random results.

Michelle's notes:

    Producing a confidence interval for area under a ROC curve: Use roc.plot in the verification library. The confidence intervals are computed by bootstrappiong the observations and predictions. There is a option called ""CI"" which should be set to TRUE, then you also need to select the number of bootstrap samples, ""n.boot"", and also the type 1 error ""alpha"" (i.e. 0.05 if you want 95% confidence intervals)."	jjr8
6	Unscheduled Release	257	The Fit GAM tool should plot bivariate smooth terms fitted with mgcv using the vis.gam function	Tools - Statistics	0.5b1	Enhancement	jjr8	assigned	2008-05-21T17:39:43-0400	2008-05-22T12:48:47-0400	"This function can produce better-looking plots than plot.gam. Michelle Sims used it to produce contour plots for our sea turtle project.

Michelle's notes:

    Produce contour or perspective plots from a bivariate smooth using gam in mgcv: Use vis.gam in the mgcv library. For example, if the model you fitted is saved as modfit, the two variables in the bivariate smooth are var1 and var2 and you wanted a contour plot, the call to options within vis.gam are: vis.gam(modfit, view=c(""var1"",""var2""), plot.type=""contour"") There are many other options you can add, one inparticular is too.far which removes points from the graph which are too far from points in the original data set. To produce a perspective plot, you simply need to replace plot.type=""persp""."	jjr8
6	Unscheduled Release	291	With ArcGIS 9.3 installed, when you use COM Automation to invoke an MGET tool that itself invokes ArcGIS, your process will crash when it exits	Core - ArcGIS Interop		Defect	jjr8	assigned	2008-10-01T10:58:12-0400	2009-06-08T20:18:22-0400	"I can reproduce this scenario in two ways. First, by running this VBScript:

{{{
#!vb
Set logger = WScript.CreateObject(""GeoEco.Logger"")
logger.Initialize()

Set hdf = WScript.CreateObject(""GeoEco.HDF"")
hdf.SDSToArcGISRaster ""C:\\MGETTest\\Conversion\\HDF\\A20050012005008.L3m_8D_SST_9.bz2"", ""C:\\MGETTest\\Conversion\\Raster\\FromHDFViaCOM\\sst"", ""l3m_data"", -180, -90, 0.0833333333333333333333, 65535, False, False, False, False, ""GEOGCS['GCS_WGS_1984',DATUM['D_WGS_1984',SPHEROID['WGS_1984',6378137.0,298.257223563]],PRIMEM['Greenwich',0.0],UNIT['Degree',0.0174532925199433]]""
}}}

When the script completes, an Application Error dialog comes up:

{{{

---------------------------
cscript.exe - Application Error
---------------------------
The instruction at ""0x734ea266"" referenced memory at ""0x1016fb84"". The memory could not be ""read"".


Click on OK to terminate the program
---------------------------
OK   
---------------------------

}}}

The second way is by running this script in R:

{{{

library(rcom)
hdf <- comCreateObject(""GeoEco.HDF"")
comInvoke(hdf, ""SDSToArcGISRaster"", ""C:\\MGETTest\\Conversion\\HDF\\A20050012005008.L3m_8D_SST_9.bz2"", ""C:\\MGETTest\\Conversion\\Raster\\FromHDFViaCOM\\sst"", ""l3m_data"", -180, -90, 0.0833333333333333333333, 65535, FALSE, FALSE, FALSE, FALSE, ""GEOGCS['GCS_WGS_1984',DATUM['D_WGS_1984',SPHEROID['WGS_1984',6378137.0,298.257223563]],PRIMEM['Greenwich',0.0],UNIT['Degree',0.0174532925199433]]"")

}}}

The script completes successfully, but when you close R, you get an identical Application Error dialog:

{{{

---------------------------
Rgui.exe - Application Error
---------------------------
The instruction at ""0x734ea266"" referenced memory at ""0x0ea5fb84"". The memory could not be ""read"".


Click on OK to terminate the program
---------------------------
OK   
---------------------------

}}}

Note that the error message ''The instruction at ""0x734ea266"" referenced memory at ""0x0ea5fb84"". The memory could not be ""read"".'' is exactly the same in the two dialog boxes. This strongly indicates that the same error is occurring.

Both of these scenarios were adapted from the [""Invoking MGET tools programmatically""] example. Both reproduce 100% with ArcGIS 9.3 installed, but do not with ArcGIS 9.2 installed. Both reproduce with pywin32 build 210 (available since 2006) and the latest build 212 (available since 2008). This suggests that ArcGIS 9.3 is the problem.

I ran the VBScript example under the debugger and got this stack trace (the deepest call is listed at the top):

{{{

>	msvbvm60.dll!RUN_INSTMGR::CPOfSink()  + 0xff bytes	
 	msvbvm60.dll!RUN_INSTMGR::DetachSink()  + 0x21 bytes	
 	msvbvm60.dll!RUN_INSTMGR::UnAdviseSinks()  + 0x7f bytes	
 	msvbvm60.dll!BASIC_CLASS::PRIVATE_UNKNOWN::Release()  + 0x161 bytes	
 	msvbvm60.dll!_BASIC_CLASS_Release@4()  + 0xd bytes	
 	msvbvm60.dll!_EVENT_SINK2_Release@4()  + 0xf bytes	
 	FdaCore.dll!711f4cc0() 	
 	[Frames below may be incorrect and/or missing, no symbols loaded for FdaCore.dll]	
 	FdaCore.dll!711ecd32() 	
 	FdaCore.dll!711ed338() 	
 	FdaCore.dll!7122a618() 	
 	FdaCore.dll!711e6621() 	
 	FdaCore.dll!7122afc6() 	
 	ole32.dll!CClassCache::CDllPathEntry::CanUnload_rl()  + 0x3f bytes	
 	ole32.dll!CClassCache::FreeUnused()  + 0x9b bytes	
 	ole32.dll!_CoFreeUnusedLibrariesEx@8()  + 0x2e bytes	
 	ole32.dll!_CoFreeUnusedLibraries@0()  + 0x9 bytes	
 	GxIMS.dll!0f63c549() 	
 	GxIMS.dll!0f68195d() 	
 	GxIMS.dll!0f681d21() 	
 	ntdll.dll!_LdrpCallInitRoutine@16()  + 0x14 bytes	
 	ntdll.dll!_LdrShutdownProcess@0()  - 0xfe bytes	
 	kernel32.dll!__ExitProcess@4()  + 0x42 bytes	
 	kernel32.dll!7c81cb0e() 	
 	cscript.exe!01001fb2() 	
 	cscript.exe!01001f8f() 	
 	kernel32.dll!_lstrlenW@4()  + 0x2d bytes	
 	kernel32.dll!_BaseProcessStart@4()  + 0x23 bytes	

}}}

This is some kind of weird COM problem, probably having to do with closing events or connection points when the process is shutting down. I have no immediate idea what to do about this. The best course of action is probably to create as small a repro scenario as possible and submit it to ESRI.

If you invoke MGET through COM Automation, watch out for this problem. As far as I know, the crash occurs only on shutdown, only with MGET tools that use ArcGIS, and only if ArcGIS 9.3 is installed. I suspect the crash will occur with any program that invokes MGET under these conditions, not just VBScript or R."	jjr8
6	Unscheduled Release	292	Create Generic OPeNDAP Fetch Tool	Tools - Conversion		Task	jjr8	assigned	2008-10-13T12:29:49-0400	2009-05-28T12:15:17-0400	OPeNDAP server configuration could be with individual Python config files, similar to the repositories specified with [http://en.wikipedia.org/wiki/Yellow_Dog_Updater_Modified yum] in the [http://trac.healthgrid.org/svn/grid-workshop/trunk/etc-yum.repos.d/ yum.repos.d].  These config files could specify the time (poch, interval) and coordinate system (based on COARDS, cdf compliance).  There is also the ability to query the attributes of the server, either on the fly at the validation step or with another tool, like HDF SDS header tool.  Then we could create a list of these server configs on an MGET server and in the ArcGIS 9.3 validation step, fetch the latest list.  Highlighting the multitude of available remote data sources on the main Wiki of the website should make MGET very attractive.	bbest
6	Unscheduled Release	293	Generating Random Pseudo-Absence Data from Extents, Lines, Grids, Etc	Tools - Spatial Analysis		Task	jjr8	assigned	2008-10-13T12:43:30-0400	2009-05-28T12:15:25-0400	"Attached is code to generate points along lines, which was initially used in the NMML MGET lab.  This brings up the larger issue of pseudo-absence sampling for generating the null data to feed into a GAM/GLM/CART/etc multivariate model.  A variety of approaches could be taken for generating these random pseudo-absences, of which the practical mechanics and statistical theory could easily constitute a worthwhile manuscript: 

 1. BBOX.  A simple bounding box.
 1. MASK.  Either a mask raster or polygon.
 1. LINE.  So the generate random points tool can work on a line.  Or use the attached file for generating even points along the line.
 1. GRID.  A grid, such as that generated by FISHNET, could be used as the presence-absence reference.  Hence intersecting segments of lines falling within the cell, and points of observation are sampled by this grid to determine presence or absence.  This is the approach taken with the SERDP project.

There remain issues of evenness when incorporating time, and suitable buffer distance in space and time to constitute an absence with respect to a presence."	bbest
6	Unscheduled Release	295	Start a Movement Toolbox	Unknown		Task	jjr8	assigned	2008-10-15T15:06:09-0400	2009-11-18T22:30:51-0500	"This is a placeholder for development of a movement toolbox, which deserves more thought.  Ideally, we could:

== Functionality ==
 * '''Import''' from data:
   * csv
   * csv with Argos' 2 sets of locations
   * Argos DIAG format
   * into a special feature class?   

 * '''Filter''' from points of data, based on:
   * location class
   * speed, distance and angle (Frietas, 2008) 
   * land
   * ...?

 * '''Generate Track''', ie line feature class, possibly with:
   * straight lines
   * berzier's spline (Tremblay, 2006)
   * ...?

 * '''Interpolate''' points along track, using:
   * probable location, using Bayesian state-space model (Jonsen, 
   * ...?

 * '''Analyse'''
   * kernel density
     * [http://nature.berkeley.edu/~alyons/locoh/ LoCoH] - Local Convex Hull Homerange Generator with web, R and ArcGIS interfaces availble ([http://www.plosone.org/article/fetchArticle.action?articleURI=info%3Adoi%2F10.1371%2Fjournal.pone.0000207 Getz et al, 2007]).
     * Other refs:
       * [http://ncg.nuim.ie/geocomputation/sessions/4B/4B1.pdf Network-Based Kernel Density Estimation for Home Range Analysis]
       * [http://www.bioone.org/doi/abs/10.2193/2006-370 Utilization Distribution Estimation Using Weighted Kernel Density Estimators]
       * [http://www.uvm.edu/~bmitchel/Publications/HR_Compare.pdf Comparison of Programs for Fixed Kernel Home Range Analysis]
   * [http://www3.interscience.wiley.com/journal/119407280/abstract?CRETRY=1&SRETRY=0 Aarts, G, M MacKenzie, B McConnell, M Fedak, et al. 2008. Estimating space-use and habitat preference from wildlife telemetry data. Ecography.]
   * habitat...
   * ...?

== R Libraries and Scripts ==
I (Ben Best) wrote some R scripts written for projects Nemo (Catherine McClellan) and Tortugas (Dana Wingfield).

 * R packages
   * [http://bm2.genes.nig.ac.jp/RGM2/pkg.php?p=argosfilter argosfilter] - speed, distance, angle filters after Freitas2008.pdf
   * [http://bm2.genes.nig.ac.jp/RGM2/pkg.php?p=trip trip] -  reading Argos data (even in original emailed DIAG format) into [sp]atial objects, calculating distances, etc
   * [http://bm2.genes.nig.ac.jp/RGM2/pkg.php?p=adehabitat adehabitat] - habitat analysis of movement data
   * ...?

 * R scripts
   * filter_argos_all.R - program to filter locations using speed, distance, angle filters from argosfilter package (Nemo)
   * filter_land.R - filters out points on land (Nemo)
   * sim_crw_v2.r - simulates a correlated random walk, sampling a given track's set of turn angles and distances, redoing if on land.  can be used as the null model for pseudo-absence data in development of a predictive model
   * ...?

== Issues ==
 * generation of pseudo-absence points for comparison
 * incorporation of surface current movement
 * ...?

== Related Work ==

 * ArcGIS tools
   * [http://www.absc.usgs.gov/glba/gistools/animal_mvmt.htm ArcView Animal Movement]
   * [http://www.spatialecology.com/htools/tooldesc.php Hawth's Tools]
   * [ARGOS-Tools http://gis-lab.info/programs/argos]
   * ...?

 * Web services
   * [http://www.seaturtle.org/stat/ Satellite Tracking Analysis Tool (STAT)]
   * ...?

 * Matlab?"	bbest
6	Unscheduled Release	301	Start building MGET for Python 2.6	Build System		Task	jjr8	assigned	2008-10-27T19:03:03-0400	2009-05-28T12:16:18-0400	Python 2.6 was released in October 2008. Few people will adopt it at first, but we need to stay ahead of the curve and start building MGET for it.	jjr8
6	Unscheduled Release	309	"Eliminate ""chunks"" parameter from Predict Bayesian Probabilities tool"	Tools - Statistics	0.6	Enhancement	jjr8	assigned	2008-11-01T10:50:17-0400	2009-05-28T12:16:31-0400	In #303, I eliminated the chunks parameter from the Predict GAM tool. I should do the same thing for the Predict Bayesian Probabilities tool, if possible	jjr8
6	Unscheduled Release	323	Rewrite the Convert Binary Raster to ArcGIS Raster tool so it doesn't read the entire binary file into memory	Tools - Conversion		Enhancement	jjr8	assigned	2009-01-14T18:28:12-0500	2009-05-28T12:16:42-0400	Global bathymetry datasets are getting quite large. For example, SRTM30_PLUS (ftp://topex.ucsd.edu/pub/srtm30_plus/topo30/) is now nearly 2 GB. The Binary to Raster tool cannot handle files this large because it is written to read them entirely into memory. I should rewrite it so it operates with a fixed buffer.	jjr8
6	Unscheduled Release	344	The Decompress Files Listed in Table and the Find and Decompress Files tools always fail	Tools - Data Management		Defect	jjr8	assigned	2009-04-18T11:13:13-0400	2009-06-08T19:42:57-0400		jjr8
6	Unscheduled Release	363	Add support for MAJORITY resampling technique to all tools that perform projections	Tools - Data Management		Enhancement	jjr8	new	2009-06-05T17:20:02-0400	2009-06-05T17:20:02-0400	ArcGIS 9.3 introduced the MAJORITY resampling technique. All MGET tools that perform projections should support it, if ArcGIS 9.3 is installed.	jjr8
6	Unscheduled Release	364	"Sample Rasters Listed in Fields fails with ArcGISError: ... Invocation of Describe(*args=(u'...',)) raised COM Error 0x80020009: ""Exception occurred.""."	Tools - Spatial Analysis		Defect	jjr8	new	2009-06-09T13:04:04-0400	2009-06-09T13:04:04-0400	"Here is a complete log output:

{{{

Executing (Sample Rasters Listed in Fields): ArcGISRasterSamplerSampleRastersInFields E:\HabModExample2\Geodatabase.mdb\PointsToSample ChlorophyllRaster;SSTRaster;SSTQualRaster Chlorophyll;SST;SSTQual # NEAREST # value;value;int(value) # E:\HabModExample2\Geodatabase.mdb\PointsToSample
Start Time: Tue Jun 09 12:08:00 2009
Running script ArcGISRasterSamplerSampleRastersInFields...
Sampling rasters: points=E:\HabModExample2\Geodatabase.mdb\PointsToSample, rasterFields=[u'ChlorophyllRaster', u'SSTRaster', u'SSTQualRaster'], sampledValueFields=[u'Chlorophyll', u'SST', u'SSTQual']...
Querying the feature class to determine the number of required sample operations.
Query complete: 0:00:01 elapsed, 600 rows retrieved, 0:00:00.002270 per row.
7 sample operations are required.
Performing sample operation 1.
ArcGISError: An ArcGIS geoprocessing function failed. This usually results from a problem with your inputs. Please check them and try again. Also review any preceding error messages and the detailed error information that appears at the end of this message. If you suspect the failure is due to a programming mistake in this tool or ArcGIS, please contact the author of this tool for assistance. Detailed error information: ArcGIS Geoprocessor object 0x00F9A350: Invocation of Describe(*args=(u'C:\\HabModExample2\\OceanographyRasters\\Chl\\8Day\\chl1999233',)) raised COM Error 0x80020009: ""Exception occurred."". Error details: source=""esriSystem.AoInitialize.1"", description=""ArcObjects has not yet been initialized"", scode=0x80004005 (""Unspecified error
""). More information may be found under topic 0 of help file ""esri_csGeoDatabase.hlp"".
The following consequences resulted from the original error:
Sampling failed.
Error in script ArcGISRasterSamplerSampleRastersInFields.
Error in executing: cmd.exe /C C:\PROGRA~1\GeoEco\ARCGIS~1\Scripts\AR88B7~1.PY  ""E:\HabModExample2\Geodatabase.mdb\PointsToSample"" ""ChlorophyllRaster;SSTRaster;SSTQualRaster"" ""Chlorophyll;SST;SSTQual"" ""#"" ""NEAREST"" ""#"" ""value;value;int(value)"" ""#"" ""E:\HabModExample2\Geodatabase.mdb\PointsToSample""

Failed to execute (Sample Rasters Listed in Fields).
End Time: Tue Jun 09 12:08:46 2009 (Elapsed Time: 46.00 seconds)

}}}

This happens on ArcGIS 9.2 SP6 when the raster that is being sampled does not exist. I consider this behavior a bug in ArcGIS; the gp.Describe function should report a better error message. To work around this, I should modify the tool to call gp.Exists prior to calling gp.Describe.

I have not tested for this specific problem with other releases of ArcGIS. They may work differently.

We should look at fixing this when we rewrite the sampling tools, ideally in MGET 0.8."	jjr8
6	Unscheduled Release	369	Add parameter to Evaluate R Statements that gives the user the option of loading tables as SP objects rather than Data Frames	Tools - Statistics		Enhancement	jjr8	assigned	2009-06-24T12:56:23-0400	2009-07-14T18:01:51-0400		jjr8
7	Unscheduled Release	12	Figure out how to obtain the ArcGIS version number on non-Windows platforms	Core - Other		Task	jjr8	reopened	2006-12-07T19:18:23-0500	2007-09-18T12:00:53-0400	This is only required when we 1) start building the Python package for non-Windows platforms, and 2) support ArcGIS on those platforms as well (required ArcGIS 9.2).	jjr8
7	Unscheduled Release	29	setup.py should fail with a nice error message if two ArcGIS tools have the same name	Build System		Task	jjr8	assigned	2007-01-08T11:58:53-0500	2007-09-18T12:01:07-0400	Currently, when a class's method is flagged as being exposed as an ArcGIS tool, CreateArcGISToolbox.exe creates a tool entry in the toolbox with the name ''!ClassName.!MethodName''. It is unlikely, but possible, that two classes with the same name might each expose a method with the same name. I'm not sure what will happen in this case; probably CreateArcGISToolbox.exe will overwrite the first tool in the toolbox with the second one. In any case, setup.py should detect this situation and report a nice error message, asking the developer to rename one of the tools.	jjr8
7	Unscheduled Release	44	Exceptions raised by classes in GeoEco.Dependencies are not easily localizable	Core - Other		Task	jjr8	assigned	2007-02-06T17:16:44-0500	2007-09-18T12:01:18-0400	Exceptions raised by classes in !GeoEco.Dependencies are not easily localizable. For example, !GeoEco.Dependencies.!PythonPackage.Initialize concatenates a bunch of sentences together, rather than using one massive format string. This will be problem if the sentences need to be reordered.	jjr8
7	Unscheduled Release	48	"Needless ""except Exception, e"" clauses should be removed."	Core - Other		Task	jjr8	assigned	2007-02-27T16:33:16-0500	2007-09-18T12:01:30-0400	"There are many places where I copy/pasted a boilerplate exception handler:

{{{
    try:
        # do something
    except Exception, e:
        # do something, but don't reference e
}}}

Most of the time I don't reference the variable e. To simplify the code and bring it in line with my intended design, these should be changed to:

{{{
    try:
        # do something
    except:
        # do something
}}}"	jjr8
7	Unscheduled Release	76	Delete obsolete HTML documentation files from TracOnlineDocumentation	Build System		Task	jjr8	assigned	2007-06-15T15:03:50-0400	2007-09-18T12:02:29-0400	The build system automatically generates HTML documentation files for new modules, classes, methods, etc. and stores them in the PythonPackage/dist/TracOnlineDocumentation tree. But it does not automatically remove HTML files when a module, class or whatever is removed from the source code. For example, the ArcGISRaster class used to have a method called !CopyRasters, but it was renamed to !CopyList. The !TracOnlineDocumentation tree still has HTML files for the !CopyRasters method. These files, and all others that refer to obsolete things, should be periodically scrubbed. Ideally the build script would automatically delete them, but this might require a call to a subversion API.	jjr8
7	Unscheduled Release	93	The process of checking for existing output ArcGIS rasters during batch processing is slow	Core - Metadata & Types	0.2	Defect	jjr8	assigned	2007-09-12T15:39:15-0400	2007-09-18T12:02:47-0400	"One user reported that it took 30 minutes to check the existence of 2700 rasters stored in his file system.

I am using ArcGIS geoprocessor functions to check the existence of ArcGIS objects such as rasters, shapefiles, etc. These functions are slow. Because ArcGIS objects can be stored in places other than the file system (e.g. databases), I can't just abandon the use of the goeprocessor functions. But I should investigate optimizations that could be used when the path to the object resolves to the local file system."	jjr8
7	Unscheduled Release	97	Add support for masking CoastWatch images using cloudx variable	Tools - Data Products - NOAA CoastWatch		Enhancement	jjr8	new	2007-09-18T18:05:16-0400	2007-09-18T18:05:16-0400	"!CoastWatch recently added a {{{cloudx}}} variable, as described in http://coastwatch.noaa.gov/helparc/sst/msg00257.html. The !CoastWatch tools should be enhanced to support masking using this variable.

It is probably not too much work to implement this, but it would touch a lot of tools and as of this writing, no users are asking for it. If demand for it develops, I will consider implementing it. Regardless, you actually can mask using the current tools by specifying the {{{cloudx}}} variable for the ""Cloud mask variable"" parameter, rather than the {{{cloud}}} variable."	jjr8
7	Unscheduled Release	98	Add option for masking the terminator in CoastWatch sst images	Tools - Data Products - NOAA CoastWatch		Task	jjr8	new	2007-09-18T19:09:13-0400	2007-09-18T19:09:13-0400	"As described in http://coastwatch.noaa.gov/helparc/sst/msg00233.html, the !CoastWatch sst images switch from a day to night equation when the solar zenith reaches 85 degrees. This manifests as ""a very faint
discontinuity down the center of the image. The discontinuity tends to
occur in cloudy data, not clear SST data."" It might be nice to provide an option to mask this, based on the sun_zenith variable.

This is really low priority, but I'm adding a ticket so at least there is a record of it."	jjr8
7	Unscheduled Release	149	"The Sample Rasters Listed in Fields tool reports duplicate warning messages: ""A NoData value was sampled for field XYZ, but you did not specify a value that should be used when NoData was sampled..."""	Tools - Spatial Analysis	0.4a2	Defect	jjr8	assigned	2007-11-30T11:24:04-0500	2009-05-28T12:17:35-0400	This message is really only supposed to be reported once. I'm not sure there is a good way around it, because Sample Rasters Listed in Fields is calling a lower level function in a loop, and that function is reporting the message. I guess I could just pass a flag back and forth so that it only gets reported the first time...	jjr8
7	Unscheduled Release	224	Tools that create ArcGIS rasters in temp directories should extract them to the final destination using ArcGISRaster.CopySilent rather than ArcGISRaster.MoveSilent	Tools - Data Management		Enhancement	jjr8	assigned	2008-03-25T09:13:07-0400	2009-05-28T12:18:45-0400	"ArcGIS does not have a ""Move"" geoprocessing tool, so MGET provides one that calls Copy followed by Delete. When a tool is extracting rasters from a temp directory, there is no need for that Delete to occur because the temp directory will be deleted shortly afterwards anyway. Tools that call Move should call Copy instead, to shave off a few tenths of a second of run time."	jjr8
7	Unscheduled Release	228	"Replace Python calls to ""for i in range(len(x))"" to ""for i, v in enumerate(x)"""	Core - Other		Task	jjr8	new	2008-03-27T21:19:17-0400	2008-03-27T21:19:17-0400	"Currently, there are probably a lot of places where this is done:

{{{
#!python
for i in range(len(x)):
    # Do something with i and x[i]
}}}

This is normally fine, but if {{{len(x)}}} is large, then the {{{range}}} call can require a lot of memory. This design:


{{{
#!python
for i, v in enumerate(x):
    # Do something with i and v
}}}

Requires less memory and is nearly as fast. For an array of 20,000,000 integers, the first design required about 7 seconds to process a certain block of code for each integer. The second design required about 7.25 seconds but only used 1/2 of the memory.

I should review the code for the first design and convert them to the second, if memory could be an issue."	jjr8
7	Unscheduled Release	282	"Update Date Formatting Help with Python Examples in Functions Like ""Sample Rasters by Field"""	Documentation		Enhancement	jjr8	assigned	2008-09-18T11:08:32-0400	2009-05-28T12:18:37-0400	"Because Python is much easier and more dynamic to format for date-time strings, especially with zero padding using C sprintf style, over Visual Basic, some useful examples might be shown for comparison in [wiki:""Sampling time series rasters""] and [http://code.env.duke.edu/projects/mget/browser/MGET/Trunk/PythonPackage/dist/TracOnlineDocumentation/Documentation/ArcGISReference/ArcGISRasterSampler.SampleRastersInFields.html?format=raw MGET Sample Rasters Listed in Fields].  For example, assuming t is a date field:

 * Python: 
   {{{
#!python
""C:/MyFolder/sst_%d%00d"" % (t[0],  t[7])
   }}}
 * Visual Basic:
   {{{
#!asp
""C:/MyFolder/sst_"" & Cstr(DatePart(""yyyy"", t)) & Right(""00"" & Cstr(DatePart(""y"", t)), 3)
   }}}

See [http://docs.python.org/lib/module-time.html Python module time] documentation for the indexing of the date object for year(0), month(1), day(2), hour(3), minute (4), second(5), weekday(6), julian day(7)."	bbest
7	Unscheduled Release	314	Add Rename Field Tool	Tools - Data Management		Task	jjr8	new	2008-11-26T16:18:45-0500	2008-11-26T16:18:45-0500	"It is surprising (and annoying) that there is no ""Rename Field"" function for Arcgis."	bbest
7	Unscheduled Release	318	Extent Tool, eg for Parameterizing Fishnet	Tools - Spatial Analysis		Enhancement	jjr8	assigned	2008-12-14T16:45:15-0500	2008-12-15T13:25:55-0500	"Needed to extract extent of a study area (eg lower-left corner) and calculate the number of rows and columns in order to use the 
[https://code.env.duke.edu/projects/mget/browser/MGET/Trunk/PythonPackage/dist/TracOnlineDocumentation/Documentation/ArcGISReference/ArcGISFishnets.CreateFishnet.html?format=raw Create Fishnet] tool, like the following output from an Albers-projected study area:

{{{
lower left corner
  x: 542081.140700
  y: 611062.131800

number of
  columns: 416
  rows: 489
}}}

The attached script could be enhanced to a simple tool for these type of outputs, which could be strung along in a !ModelBuilder window as parameters to tools like Create Fishnet."	bbest
7	Unscheduled Release	319	Create Fishnet tool progress should say Cells, not Rows	Tools - Spatial Analysis		Task	jjr8	assigned	2008-12-14T17:15:33-0500	2009-05-28T12:20:10-0400	"The progress of output should say ""cells"" not ""rows"" in output below:

{{{
Start Time: Sun Dec 14 16:46:34 2008
Running script ArcGISFishnetsCreateFishnet...
Inserting fishnet cells.
Still inserting rows: 0:01:00 elapsed, 10480 rows inserted, 0:00:00.005725 per row.
Still inserting rows: 0:06:00 elapsed, 62271 rows inserted, 0:00:00.005781 per row.
Still inserting rows: 0:11:00 elapsed, 114005 rows inserted, 0:00:00.005789 per row.
Still inserting rows: 0:16:00 elapsed, 165705 rows inserted, 0:00:00.005793 per row.
Finished inserting rows: 0:19:39 elapsed, 203424 rows inserted, 0:00:00.005796 per row.
Clipping the fishnet to the study area features.
Calculating cell areas.
Update in progress: 0:01:00 elapsed, 32886 rows updated, 0 deleted, 0 unchanged, 0:00:00.001824 per row, 32580 remaining, estimated completion time: 5:09:41 PM.
Update complete: 0:01:59 elapsed, 65466 rows updated, 0 deleted, 0 unchanged, 0:00:00.001824 per row.
Completed script ArcGISFishnetsCreateFishnet...
Executed (Create Fishnet (4)) successfully.
End Time: Sun Dec 14 17:09:58 2008 (Elapsed Time: 23 minutes 24 seconds)
}}}

This is confirmed by the number of columns and rows specified in the tool:
{{{
>>> ncols*nrows
203424
>>> ncols*nrows * 0.005796 / 60
19.650758399999997
}}}

When inserting fishnet cells, it would also be nice to see the number of cells remaining and estimated completion time, like with the area calculation."	bbest
7	Unscheduled Release	560	Add Utility: Table to CSV	Tools - Data Management		Enhancement	jjr8	new	2013-04-15T14:58:35-0400	2013-04-15T14:58:35-0400	"Kinda silly that this isn't already an option. Here's a function to handle funky characters and check for quotes to use quote delimited strings per preferred Excel format.


{{{
#!python
# arcpy environment
import arcpy, codecs

# get paths
tbl = arcpy.GetParameterAsText(0)
csv = arcpy.GetParameterAsText(1)

def tbl2csv(tbl, csv):
    #tbl ='regions'; csv = r'D:\best\ez\data\csv\regions.csv'
    fields = [f for f in arcpy.ListFields(tbl) if f.name not in ['Shape','Shape_Area','Shape_Length','FID']]
    field_names = [fld.name for fld in fields]
    f = open(csv, 'wb')
    f.write(codecs.BOM_UTF8)
    f.write(','.join(field_names)+'\n') #csv headers
    cursor = arcpy.da.SearchCursor(tbl, field_names)
    for i,row in enumerate(cursor):
        for j,fld in enumerate(fields):
            if j > 0:
                f.write(',')
            if fld.type in ['String']:
                if repr(row[j]).find('""')> 0:
                    raise ValueError('Quote character("") found in row' + repr(row) + '\n  with headers' + repr(field_names))
                else:
                    f.write(unicode('""' + row[j] + '""').encode(""utf-8""))
            else:
                f.write(repr(row[j]))
        f.write('\n')          
    f.close(); del cursor, f

tbl2csv(tbl, csv)
}}}"	bbest
7	Unscheduled Release	561	Add Utility: Rename Field	Tools - Data Management		Enhancement	jjr8	new	2013-04-15T15:03:45-0400	2013-04-15T15:04:38-0400	"Inability to simply rename a field is a regular frustration with ArcGIS. Here's a little start on code to do this with default options for the given data type (eg field_length, field_precision).


{{{
#!python
import arcpy

# get paths
tbl = arcpy.GetParameterAsText(0)
fld_fro = arcpy.GetParameterAsText(1)
fld_to = arcpy.GetParameterAsText(2)

def rename_field(tbl, fld_fro, fld_to):
    if fld_fro not in [fld.name for fld in arcpy.ListFields(tbl)]:
        raise NameError(""Field '%s' not found in table '%s'"" % (fld_fro, tbl))
    if fld_to in [fld.name for fld in arcpy.ListFields(tbl)]:
        raise NameError(""Field '%s' already exists in table '%s'"" % (fld_fro, tbl))
    fld_type = [fld.type for fld in arcpy.ListFields(tbl) if fld.name == fld_fro][0]
    # TODO: figure out rest of field information, eg field_length, field_precision, etc.
    arcpy.AddField_management(tbl, fld_to, fld_type)
    arcpy.CalculateField_management(tbl, fld_to, ""!%s!"" % fld_fro, ""PYTHON_9.3"")
    arcpy.DeleteField_management(tbl, fld_fro)    

rename_field(tbl, fld_fro, fld_to)
}}}"	bbest