[Therion] Scrap limits

[Benedikt Hallinger]

Hello,
for further testing i wrote a small script generating a sample cave of 
variable length.

The bash-script generateTestCave.sh generates a th-file one main survey and 
*n* linked subusrveys with included scrap and is called as following:
   ./generateTestCave.sh <number-of-surveys> test.th
eg "./generateTestCave.sh 1000 test.th" generates a survey "test" containing 
1000 linked subsurveys containing each a suitable inline-copy of the 
scrap.th. On the main "test"-survey level also for each subsurvey a "map 
n-Plan" is generated.
This is then selected in the "thconfig" file.

With the contained thconig file one can pay a little to explore behavior.

What i encounter is the following unexepected:
With raising survey count the memory needed and time to compile increases 
very heavily. This is DESPITE i only select 3 distinct maps for output in lox 
and pdf plan projection.
Why is that so? i expected, that the selection influences the scraps 
selected and thus compile time should nearly remain constant (at least only 
increase slightly due to more data processed and discarded)?
Also, in the mailing list there was an encounter of strange behavior of the 
select/unselect command - do we have a bug here somewhere that also causes 
the behavior i see here?




Am 2017-12-03 10:41, schrieb Benedikt Hallinger:
> Thank, how could i switch this?
> I also wrote a small testcase that segfaulted at about 3000 scraps.
>
> Am 2017-12-02 9:55, schrieb Martin Budaj via Therion:
>> Hi,
>>
>> there is no change regarding the limits in Therion. If there is a real
>> need, following could be done:
>>
>> - in the current version of MetaPost, it's possible to used "double"
>> arithmetic just by specifying a command line option, which practically
>> eliminates MetaPost limits
>>
>> - instead of pdfTeX we could use LuaTeX to produce the PDFs. This
>> doubles the number of registers available from 32768 to 65536.
>> Registers are needed to reference the fragments of all
>> scraps/sections; you usually need up to 6 of them for a scrap. So you
>> would get maybe 12000 instead of 6000 scraps in one output file. It
>> would require some substantial work to support LuaTeX (there is e.g.
>> completely different font handling compared to PdfTeX).
>>
>> And yes, the limit applies just to the data selected for export.
>>
>> Martin
>>
>>
>> On Tue, Nov 28, 2017 at 9:44 PM, Benedikt Hallinger via Therion
>> <therion at speleo.sk> wrote:
>>> Maybe another question:
>>> Assume a large cave with thousands of scraps.
>>> When i make a thconfig file sourcing all that data, but using "select"
>>> statements i only select partial data,
>>> does the metapost limit apply to the whole dataset or just the scraps
>>> covered by the select command?
>>>
>>>
>>>
>>> Am 2017-11-28 22:19, schrieb Benedikt Hallinger via Therion:
>>>>
>>>> Hello Martin,
>>>> is the blow limit of 4096 scraps still valid in the current version?
>>>> Or is it already fixed so we can use more scraps?
>>>>
>>>>
>>>>
>>>>> On Tue, Dec 1, 2009 at 5:26 PM, Carl Magnuson <magnu213 at umn.edu>
>>>>> wrote:
>>>>>>
>>>>>> It looks like the solution is to issue the following metapost 
>>>>>> command:
>>>>>> warningcheck := 0;
>>>>>
>>>>>
>>>>> Indeed. The new limit will be 32768 and could not be increased further
>>>>> in Metapost itself.
>>>>>
>>>>> The solution would be modification of how therion manages metapost
>>>>> pictures (currently they are stored in files data.1 to data.4000, with
>>>>> files data.4001 to data.4095 reserved for pattern definitions). This
>>>>> numbering scheme could be modified to allow more file name prefixes
>>>>> and consequently theoretically unlimited number of scraps processed by
>>>>> metapost.
>>>>>
>>>>> On the other hand there is still pdfTeX limit which would not allow
>>>>> much more scraps. PdfTeX uses internal registers for scraps
>>>>> referencing (scrap data is included only once in pdf file and can be
>>>>> referenced on multiple pages). You could avoid pdftex limit by using
>>>>> SVG output (if SVG viewers would process large number of internal
>>>>> references).
>>>>>
>>>>> In the longer-term future (a few years) I would like to use metapost
>>>>> as a library instead of external metapost executable, which would
>>>>> solve the problems with temporary files (and other problems as well).
>>>>>
>>>>>> However adding it in a
>>>>>> code metapost
>>>>>> warningcheck := 0;
>>>>>> endcode
>>>>>> block seems to have no effect, mpost still fails on more then 4096
>>>>>> scraps.
>>>>>
>>>>>
>>>>> Therion currently inserts warningcheck:=1; before scraps without good
>>>>> reason, so it will be fixed soon.
>>>>>
>>>>> If the new warningcheck setting would work for you, I would prefer not
>>>>> to modify current file numbering scheme for metapost pictures and have
>>>>> it fixed later with implementation of metapost library.
>>>>>
>>>>> Martin
>>>>
>>>> _______________________________________________
>>>> Therion mailing list
>>>> Therion at speleo.sk
>>>> https://mailman.speleo.sk/listinfo/therion
>>>
>>>
>>> _______________________________________________
>>> Therion mailing list
>>> Therion at speleo.sk
>>> https://mailman.speleo.sk/listinfo/therion
>> _______________________________________________
>> Therion mailing list
>> Therion at speleo.sk
>> https://mailman.speleo.sk/listinfo/therion
-------------- next part --------------
A non-text attachment was scrubbed...
Name: masstest.tar.gz
Type: application/octet-stream
Size: 459513 bytes
Desc: not available
URL: <http://mailman.speleo.sk/pipermail/therion/attachments/20171227/09a9eab6/attachment.obj>