[Therion] Scrap limits
Benedikt Hallinger
beni at hallinger.org
Fri Dec 29 15:42:08 CET 2017
Shouldnt they be placed at the actual position according to station
coordinates prior joining, ie the scrap point reference coordinates be local
to its namespace?
Am 2017-12-29 11:20, schrieb Stacho Mudrak via Therion:
> Hello, I have not investigated to details a lot, but the problem with
> this huge file is search for automatic joins. All the scraps are in one
> giant file and have a lot of common points so it tries to auto-connect every
> scrap with every scrap, therefore n^2 memory usage. Situation that should
> not happen in reality.
>
> If you apply attached patch and disable automatic scrap joins - therion
> runs smoothly even with such a giant structure. But more proper solution
> would be to shift every scrap, that they will not have common points.
>
> S.
>
> diff --git a/thdb2d.cxx b/thdb2d.cxx
> index b8c9720..7d7b72e 100644
> --- a/thdb2d.cxx
> +++ b/thdb2d.cxx
> @@ -2428,6 +2428,7 @@ void thdb2d::pp_process_joins(thdb2dprj * prj)
>
> // prida na koniec automaticke joiny
> jci = joincandlist.begin();
> + jci = joincandlist.end();
> while (jci != joincandlist.end()) {
> njci = jci;
> njci++;
>
>
>
> On 27 December 2017 at 13:26, Benedikt Hallinger via Therion
> <therion at speleo.sk [14]> wrote:
>
>> Hello,
>> for further testing i wrote a small script generating a sample cave of
>> variable length.
>>
>> The bash-script generateTestCave.sh generates a th-file one main survey
>> and *n* linked subusrveys with included scrap and is called as following:
>> ./generateTestCave.sh <number-of-surveys> test.th [9]
>> eg "./generateTestCave.sh 1000 test.th [10]" generates a survey "test"
>> containing 1000 linked subsurveys containing each a suitable inline-copy of
>> the scrap.th [11]. On the main "test"-survey level also for each subsurvey
>> a "map n-Plan" is generated.
>> This is then selected in the "thconfig" file.
>>
>> With the contained thconig file one can pay a little to explore behavior.
>>
>> What i encounter is the following unexepected:
>> With raising survey count the memory needed and time to compile increases
>> very heavily. This is DESPITE i only select 3 distinct maps for output in
>> lox and pdf plan projection.
>> Why is that so? i expected, that the selection influences the scraps
>> selected and thus compile time should nearly remain constant (at least only
>> increase slightly due to more data processed and discarded)?
>> Also, in the mailing list there was an encounter of strange behavior of
>> the select/unselect command - do we have a bug here somewhere that also
>> causes the behavior i see here?
>>
>> Am 2017-12-03 10:41, schrieb Benedikt Hallinger:
>>
>>> Thank, how could i switch this?
>>> I also wrote a small testcase that segfaulted at about 3000 scraps.
>>>
>>> Am 2017-12-02 9:55, schrieb Martin Budaj via Therion:
>>>
>>>> Hi,
>>>>
>>>> there is no change regarding the limits in Therion. If there is a real
>>>> need, following could be done:
>>>>
>>>> - in the current version of MetaPost, its possible to used "double"
>>>> arithmetic just by specifying a command line option, which practically
>>>> eliminates MetaPost limits
>>>>
>>>> - instead of pdfTeX we could use LuaTeX to produce the PDFs. This
>>>> doubles the number of registers available from 32768 to 65536.
>>>> Registers are needed to reference the fragments of all
>>>> scraps/sections; you usually need up to 6 of them for a scrap. So you
>>>> would get maybe 12000 instead of 6000 scraps in one output file. It
>>>> would require some substantial work to support LuaTeX (there is e.g.
>>>> completely different font handling compared to PdfTeX).
>>>>
>>>> And yes, the limit applies just to the data selected for export.
>>>>
>>>> Martin
>>>>
>>>> On Tue, Nov 28, 2017 at 9:44 PM, Benedikt Hallinger via Therion
>>>> <therion at speleo.sk [6]> wrote:
>>>>
>>>>> Maybe another question:
>>>>> Assume a large cave with thousands of scraps.
>>>>> When i make a thconfig file sourcing all that data, but using "select"
>>>>> statements i only select partial data,
>>>>> does the metapost limit apply to the whole dataset or just the scraps
>>>>> covered by the select command?
>>>>>
>>>>> Am 2017-11-28 22:19, schrieb Benedikt Hallinger via Therion:
>>>>>
>>>>>> Hello Martin,
>>>>>> is the blow limit of 4096 scraps still valid in the current version?
>>>>>> Or is it already fixed so we can use more scraps?
>>>>>>
>>>>>>> On Tue, Dec 1, 2009 at 5:26 PM, Carl Magnuson <magnu213 at umn.edu
>>>>>>> [1]>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> It looks like the solution is to issue the following metapost
>>>>>>>> command:
>>>>>>>> warningcheck := 0;
>>>>>>>
>>>>>>> Indeed. The new limit will be 32768 and could not be increased
>>>>>>> further
>>>>>>> in Metapost itself.
>>>>>>>
>>>>>>> The solution would be modification of how therion manages metapost
>>>>>>> pictures (currently they are stored in files data.1 to data.4000,
>>>>>>> with
>>>>>>> files data.4001 to data.4095 reserved for pattern definitions). This
>>>>>>> numbering scheme could be modified to allow more file name prefixes
>>>>>>> and consequently theoretically unlimited number of scraps processed
>>>>>>> by
>>>>>>> metapost.
>>>>>>>
>>>>>>> On the other hand there is still pdfTeX limit which would not allow
>>>>>>> much more scraps. PdfTeX uses internal registers for scraps
>>>>>>> referencing (scrap data is included only once in pdf file and can be
>>>>>>> referenced on multiple pages). You could avoid pdftex limit by using
>>>>>>> SVG output (if SVG viewers would process large number of internal
>>>>>>> references).
>>>>>>>
>>>>>>> In the longer-term future (a few years) I would like to use metapost
>>>>>>> as a library instead of external metapost executable, which would
>>>>>>> solve the problems with temporary files (and other problems as
>>>>>>> well).
>>>>>>>
>>>>>>>> However adding it in a
>>>>>>>> code metapost
>>>>>>>> warningcheck := 0;
>>>>>>>> endcode
>>>>>>>> block seems to have no effect, mpost still fails on more then 4096
>>>>>>>> scraps.
>>>>>>>
>>>>>>> Therion currently inserts warningcheck:=1; before scraps without
>>>>>>> good
>>>>>>> reason, so it will be fixed soon.
>>>>>>>
>>>>>>> If the new warningcheck setting would work for you, I would prefer
>>>>>>> not
>>>>>>> to modify current file numbering scheme for metapost pictures and
>>>>>>> have
>>>>>>> it fixed later with implementation of metapost library.
>>>>>>>
>>>>>>> Martin
>>>>>>
>>>>>> _______________________________________________
>>>>>> Therion mailing list
>>>>>> Therion at speleo.sk [2]
>>>>>> https://mailman.speleo.sk/listinfo/therion [3]
>>>>>
>>>>> _______________________________________________
>>>>> Therion mailing list
>>>>> Therion at speleo.sk [4]
>>>>> https://mailman.speleo.sk/listinfo/therion [5]
>>>> _______________________________________________
>>>> Therion mailing list
>>>> Therion at speleo.sk [7]
>>>> https://mailman.speleo.sk/listinfo/therion [8]
>>
>> _______________________________________________
>> Therion mailing list
>> Therion at speleo.sk [12]
>> https://mailman.speleo.sk/listinfo/therion [13]
>
>
>
> Links:
> ------
> [1] http://umn.edu
> [2] mailto:Therion at speleo.sk
> [3] https://mailman.speleo.sk/listinfo/therion
> [4] mailto:Therion at speleo.sk
> [5] https://mailman.speleo.sk/listinfo/therion
> [6] mailto:therion at speleo.sk
> [7] mailto:Therion at speleo.sk
> [8] https://mailman.speleo.sk/listinfo/therion
> [9] http://test.th
> [10] http://test.th
> [11] http://scrap.th
> [12] mailto:Therion at speleo.sk
> [13] https://mailman.speleo.sk/listinfo/therion
> [14] mailto:therion at speleo.sk
More information about the Therion
mailing list