[Therion] Scrap limits

Benedikt Hallinger beni at hallinger.org
Sat Dec 30 17:26:23 CET 2017 

Hello,
i played a little more and it looks good.
For the archive, here the source of the revised script.
I changed it so each survey gets its own th2-file now. Included is now also 
an extendet elevation and two sectionals.

-----------------------------------
tar -xzf MASSTEST.tar.gz
cd MASSTEST
./generateTestCave.sh 15000  #bash; produces 15k surveys (~250km) and scrap 
data in out/
therion    # results are in results/
-----------------------------------


I tested atlas on more data (see thconfig-more):
Can i unselect the sectionals to get more room for plan-data?
Using "symbol-hide point section" in layout seems to not conserve any 
memory...
The limit with this dataset seems to be somewhere between 1350 and 1380 
selected maps (1350 still compiles successfully; each survey contains one 
plan- and one none-scrap (section); total 2).


With best regards and thank you!



Am 2017-12-29 22:47, schrieb Stacho Mudrak via Therion:
> I am sorry, I am not sure I understand.
>
> The problem is, if there are two lines in two different scraps within one 
> .th2 file starting or ending from point with same coordinates, therion 
> automatically creates a join between them. In your file, each of line of 
> 5000 scraps are the same. Therefore they start and end at same points. 
> Therefore therion automatically creates 5000x5000 joins for each line end. 
> And this is the problem. I have modified your script to add some randomness 
> to the coordinates (just shift every scrap by random vector) and it solves 
> the problem.
>
> S.
>
> On 29 December 2017 at 15:42, Benedikt Hallinger via Therion 
> <therion at speleo.sk [31]> wrote:
>
>> Shouldnt they be placed at the actual position according to station 
>> coordinates prior joining, ie the scrap point reference coordinates be 
>> local to its namespace?
>>
>> Am 2017-12-29 11:20, schrieb Stacho Mudrak via Therion:
>>
>>> ​Hello, I have not investigated to details a lot, but the problem with 
>>> this huge file is search for automatic joins. All the scraps are in one 
>>> giant file and have a lot of common points so it tries to auto-connect 
>>> every scrap with every scrap, therefore n^2 memory usage. Situation that 
>>> should not happen in reality.
>>>
>>> If you apply attached patch and disable automatic scrap joins - therion 
>>> runs smoothly even with such a giant structure. But more proper solution 
>>> would be to shift every scrap, that they will not have common points.
>>>
>>> S.
>>>
>>> diff --git a/thdb2d.cxx b/thdb2d.cxx
>>> index b8c9720..7d7b72e 100644
>>> --- a/thdb2d.cxx
>>> +++ b/thdb2d.cxx
>>> @@ -2428,6 +2428,7 @@ void thdb2d::pp_process_joins(thdb2dprj * prj)
>>>   
>>>    // prida na koniec automaticke joiny
>>>    jci = joincandlist.begin();
>>> +  jci = joincandlist.end();
>>>    while (jci != joincandlist.end()) {
>>>      njci = jci;
>>>      njci++;
>>>
>>> ​
>>>
>>> On 27 December 2017 at 13:26, Benedikt Hallinger via Therion 
>>> <therion at speleo.sk [14] [14]> wrote:
>>>
>>>> Hello,
>>>> for further testing i wrote a small script generating a sample cave of 
>>>> variable length.
>>>>
>>>> The bash-script generateTestCave.sh generates a th-file one main survey 
>>>> and *n* linked subusrveys with included scrap and is called as following:
>>>>   ./generateTestCave.sh <number-of-surveys> test.th [9] [9]
>>>> eg "./generateTestCave.sh 1000 test.th [10] [10]" generates a survey 
>>>> "test" containing 1000 linked subsurveys containing each a suitable 
>>>> inline-copy of the scrap.th [11] [11]. On the main "test"-survey level 
>>>> also for each subsurvey a "map n-Plan" is generated.
>>>> This is then selected in the "thconfig" file.
>>>>
>>>> With the contained thconig file one can pay a little to explore 
>>>> behavior.
>>>>
>>>> What i encounter is the following unexepected:
>>>> With raising survey count the memory needed and time to compile 
>>>> increases very heavily. This is DESPITE i only select 3 distinct maps for 
>>>> output in lox and pdf plan projection.
>>>> Why is that so? i expected, that the selection influences the scraps 
>>>> selected and thus compile time should nearly remain constant (at least 
>>>> only increase slightly due to more data processed and discarded)?
>>>> Also, in the mailing list there was an encounter of strange behavior of 
>>>> the select/unselect command - do we have a bug here somewhere that also 
>>>> causes the behavior i see here?
>>>>
>>>> Am 2017-12-03 10:41, schrieb Benedikt Hallinger:
>>>>
>>>>> Thank, how could i switch this?
>>>>> I also wrote a small testcase that segfaulted at about 3000 scraps.
>>>>>
>>>>> Am 2017-12-02 9:55, schrieb Martin Budaj via Therion:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> there is no change regarding the limits in Therion. If there is a 
>>>>>> real
>>>>>> need, following could be done:
>>>>>>
>>>>>> - in the current version of MetaPost, its possible to used "double"
>>>>>> arithmetic just by specifying a command line option, which 
>>>>>> practically
>>>>>> eliminates MetaPost limits
>>>>>>
>>>>>> - instead of pdfTeX we could use LuaTeX to produce the PDFs. This
>>>>>> doubles the number of registers available from 32768 to 65536.
>>>>>> Registers are needed to reference the fragments of all
>>>>>> scraps/sections; you usually need up to 6 of them for a scrap. So you
>>>>>> would get maybe 12000 instead of 6000 scraps in one output file. It
>>>>>> would require some substantial work to support LuaTeX (there is e.g.
>>>>>> completely different font handling compared to PdfTeX).
>>>>>>
>>>>>> And yes, the limit applies just to the data selected for export.
>>>>>>
>>>>>> Martin
>>>>>>
>>>>>> On Tue, Nov 28, 2017 at 9:44 PM, Benedikt Hallinger via Therion
>>>>>> <therion at speleo.sk [6] [6]> wrote:
>>>>>>
>>>>>>> Maybe another question:
>>>>>>> Assume a large cave with thousands of scraps.
>>>>>>> When i make a thconfig file sourcing all that data, but using 
>>>>>>> "select"
>>>>>>> statements i only select partial data,
>>>>>>> does the metapost limit apply to the whole dataset or just the 
>>>>>>> scraps
>>>>>>> covered by the select command?
>>>>>>>
>>>>>>> Am 2017-11-28 22:19, schrieb Benedikt Hallinger via Therion:
>>>>>>>
>>>>>>>> Hello Martin,
>>>>>>>> is the blow limit of 4096 scraps still valid in the current 
>>>>>>>> version?
>>>>>>>> Or is it already fixed so we can use more scraps?
>>>>>>>>
>>>>>>>>> On Tue, Dec 1, 2009 at 5:26 PM, Carl Magnuson <magnu213 at umn.edu 
>>>>>>>>> [1] [1]>
>>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> It looks like the solution is to issue the following metapost 
>>>>>>>>>> command:
>>>>>>>>>> warningcheck := 0;
>>>>>>>>>
>>>>>>>>> Indeed. The new limit will be 32768 and could not be increased 
>>>>>>>>> further
>>>>>>>>> in Metapost itself.
>>>>>>>>>
>>>>>>>>> The solution would be modification of how therion manages metapost
>>>>>>>>> pictures (currently they are stored in files data.1 to data.4000, 
>>>>>>>>> with
>>>>>>>>> files data.4001 to data.4095 reserved for pattern definitions). 
>>>>>>>>> This
>>>>>>>>> numbering scheme could be modified to allow more file name 
>>>>>>>>> prefixes
>>>>>>>>> and consequently theoretically unlimited number of scraps 
>>>>>>>>> processed by
>>>>>>>>> metapost.
>>>>>>>>>
>>>>>>>>> On the other hand there is still pdfTeX limit which would not 
>>>>>>>>> allow
>>>>>>>>> much more scraps. PdfTeX uses internal registers for scraps
>>>>>>>>> referencing (scrap data is included only once in pdf file and can 
>>>>>>>>> be
>>>>>>>>> referenced on multiple pages). You could avoid pdftex limit by 
>>>>>>>>> using
>>>>>>>>> SVG output (if SVG viewers would process large number of internal
>>>>>>>>> references).
>>>>>>>>>
>>>>>>>>> In the longer-term future (a few years) I would like to use 
>>>>>>>>> metapost
>>>>>>>>> as a library instead of external metapost executable, which would
>>>>>>>>> solve the problems with temporary files (and other problems as 
>>>>>>>>> well).
>>>>>>>>>
>>>>>>>>>> However adding it in a
>>>>>>>>>> code metapost
>>>>>>>>>> warningcheck := 0;
>>>>>>>>>> endcode
>>>>>>>>>> block seems to have no effect, mpost still fails on more then 
>>>>>>>>>> 4096
>>>>>>>>>> scraps.
>>>>>>>>>
>>>>>>>>> Therion currently inserts warningcheck:=1; before scraps without 
>>>>>>>>> good
>>>>>>>>> reason, so it will be fixed soon.
>>>>>>>>>
>>>>>>>>> If the new warningcheck setting would work for you, I would prefer 
>>>>>>>>> not
>>>>>>>>> to modify current file numbering scheme for metapost pictures and 
>>>>>>>>> have
>>>>>>>>> it fixed later with implementation of metapost library.
>>>>>>>>>
>>>>>>>>> Martin
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Therion mailing list
>>>>>>>> Therion at speleo.sk [2] [2]
>>>>>>>> https://mailman.speleo.sk/listinfo/therion [3] [3]
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Therion mailing list
>>>>>>> Therion at speleo.sk [4] [4]
>>>>>>> https://mailman.speleo.sk/listinfo/therion [5] [5]
>>>>>> _______________________________________________
>>>>>> Therion mailing list
>>>>>> Therion at speleo.sk [7] [7]
>>>>>> https://mailman.speleo.sk/listinfo/therion [8] [8]
>>>>
>>>> _______________________________________________
>>>> Therion mailing list
>>>> Therion at speleo.sk [12] [12]
>>>> https://mailman.speleo.sk/listinfo/therion [13] [13]
>>>
>>> Links:
>>> ------
>>> [1] http://umn.edu [15]
>>> [2] mailto:Therion at speleo.sk [16]
>>> [3] https://mailman.speleo.sk/listinfo/therion [17]
>>> [4] mailto:Therion at speleo.sk [18]
>>> [5] https://mailman.speleo.sk/listinfo/therion [19]
>>> [6] mailto:therion at speleo.sk [20]
>>> [7] mailto:Therion at speleo.sk [21]
>>> [8] https://mailman.speleo.sk/listinfo/therion [22]
>>> [9] http://test.th [23]
>>> [10] http://test.th [24]
>>> [11] http://scrap.th [25]
>>> [12] mailto:Therion at speleo.sk [26]
>>> [13] https://mailman.speleo.sk/listinfo/therion [27]
>>> [14] mailto:therion at speleo.sk [28]
>>
>> _______________________________________________
>> Therion mailing list
>> Therion at speleo.sk [29]
>> https://mailman.speleo.sk/listinfo/therion [30]
>
>
>
> Links:
> ------
> [1] http://umn.edu
> [2] mailto:Therion at speleo.sk
> [3] https://mailman.speleo.sk/listinfo/therion
> [4] mailto:Therion at speleo.sk
> [5] https://mailman.speleo.sk/listinfo/therion
> [6] mailto:therion at speleo.sk
> [7] mailto:Therion at speleo.sk
> [8] https://mailman.speleo.sk/listinfo/therion
> [9] http://test.th
> [10] http://test.th
> [11] http://scrap.th
> [12] mailto:Therion at speleo.sk
> [13] https://mailman.speleo.sk/listinfo/therion
> [14] mailto:therion at speleo.sk
> [15] http://umn.edu
> [16] mailto:Therion at speleo.sk
> [17] https://mailman.speleo.sk/listinfo/therion
> [18] mailto:Therion at speleo.sk
> [19] https://mailman.speleo.sk/listinfo/therion
> [20] mailto:therion at speleo.sk
> [21] mailto:Therion at speleo.sk
> [22] https://mailman.speleo.sk/listinfo/therion
> [23] http://test.th
> [24] http://test.th
> [25] http://scrap.th
> [26] mailto:Therion at speleo.sk
> [27] https://mailman.speleo.sk/listinfo/therion
> [28] mailto:therion at speleo.sk
> [29] mailto:Therion at speleo.sk
> [30] https://mailman.speleo.sk/listinfo/therion
> [31] mailto:therion at speleo.sk
-------------- next part --------------
A non-text attachment was scrubbed...
Name: MASSTEST.tar.gz
Type: application/octet-stream
Size: 3017 bytes
Desc: not available
URL: <http://mailman.speleo.sk/pipermail/therion/attachments/20171230/ac0bb34a/attachment.obj>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: thconfig-more
URL: <http://mailman.speleo.sk/pipermail/therion/attachments/20171230/ac0bb34a/attachment.ksh>

More information about the Therion mailing list