However, some algorithms do not lend themselves to this, so we leave it alone. If we are willing to violate rule number three in the original post, another 7/8 of the time can be reduced by switching to a fully numeric processing. More than half of the original time (68 -> 27 sec) was consumed with inefficiencies in the str2num call, which can be removed by switching the sscanf.Ībout another 2/3 of the remaining time (27 -> 8 sec) can be reduced by using larger batches for both file reading and string to number conversions. Fully batched operations (non-compliant).Using java single line file reader and sscanf on single lines.Reading large batches into memory, then sscanf.I put together a quick script to test out the ingestion speed (and consistency of result) of 6 variations on these themes. making the actual processing more efficient (which I have ruled out via rule 3, above).performing the string to number conversion more efficiently (either via batching, or using better functions). With that in mind, the answers and comments seem to be encouraging efficiency in three areas: The actual operation (what the OP calls "do stuff with nums") must be performed one row at a time, cannot be vectorized. The method must scale to reading files that are too large to be contained in memory, (although my patience is limited, so my test file is only 500,000 lines). Here are my assumptions:Ī well formatted ASCII file, containing two columns of numbers.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |