[Therion] Scrap limits
Stacho Mudrak
lists at group-s.sk
Fri Dec 29 10:20:14 CET 2017
Hello, I have not investigated to details a lot, but the problem with this
huge file is search for automatic joins. All the scraps are in one giant
file and have a lot of common points so it tries to auto-connect every
scrap with every scrap, therefore n^2 memory usage. Situation that should
not happen in reality.
If you apply attached patch and disable automatic scrap joins - therion
runs smoothly even with such a giant structure. But more proper solution
would be to shift every scrap, that they will not have common points.
S.
diff --git a/thdb2d.cxx b/thdb2d.cxx
index b8c9720..7d7b72e 100644
--- a/thdb2d.cxx
+++ b/thdb2d.cxx
@@ -2428,6 +2428,7 @@ void thdb2d::pp_process_joins(thdb2dprj * prj)
// prida na koniec automaticke joiny
jci = joincandlist.begin();
+ jci = joincandlist.end();
while (jci != joincandlist.end()) {
njci = jci;
njci++;
On 27 December 2017 at 13:26, Benedikt Hallinger via Therion <
therion at speleo.sk> wrote:
> Hello,
> for further testing i wrote a small script generating a sample cave of
> variable length.
>
> The bash-script generateTestCave.sh generates a th-file one main survey
> and *n* linked subusrveys with included scrap and is called as following:
> ./generateTestCave.sh <number-of-surveys> test.th
> eg "./generateTestCave.sh 1000 test.th" generates a survey "test"
> containing 1000 linked subsurveys containing each a suitable inline-copy of
> the scrap.th. On the main "test"-survey level also for each subsurvey a
> "map n-Plan" is generated.
> This is then selected in the "thconfig" file.
>
> With the contained thconig file one can pay a little to explore behavior.
>
> What i encounter is the following unexepected:
> With raising survey count the memory needed and time to compile increases
> very heavily. This is DESPITE i only select 3 distinct maps for output in
> lox and pdf plan projection.
> Why is that so? i expected, that the selection influences the scraps
> selected and thus compile time should nearly remain constant (at least only
> increase slightly due to more data processed and discarded)?
> Also, in the mailing list there was an encounter of strange behavior of
> the select/unselect command - do we have a bug here somewhere that also
> causes the behavior i see here?
>
>
>
>
>
> Am 2017-12-03 10:41, schrieb Benedikt Hallinger:
>
>> Thank, how could i switch this?
>> I also wrote a small testcase that segfaulted at about 3000 scraps.
>>
>> Am 2017-12-02 9:55, schrieb Martin Budaj via Therion:
>>
>>> Hi,
>>>
>>> there is no change regarding the limits in Therion. If there is a real
>>> need, following could be done:
>>>
>>> - in the current version of MetaPost, it's possible to used "double"
>>> arithmetic just by specifying a command line option, which practically
>>> eliminates MetaPost limits
>>>
>>> - instead of pdfTeX we could use LuaTeX to produce the PDFs. This
>>> doubles the number of registers available from 32768 to 65536.
>>> Registers are needed to reference the fragments of all
>>> scraps/sections; you usually need up to 6 of them for a scrap. So you
>>> would get maybe 12000 instead of 6000 scraps in one output file. It
>>> would require some substantial work to support LuaTeX (there is e.g.
>>> completely different font handling compared to PdfTeX).
>>>
>>> And yes, the limit applies just to the data selected for export.
>>>
>>> Martin
>>>
>>>
>>> On Tue, Nov 28, 2017 at 9:44 PM, Benedikt Hallinger via Therion
>>> <therion at speleo.sk> wrote:
>>>
>>>> Maybe another question:
>>>> Assume a large cave with thousands of scraps.
>>>> When i make a thconfig file sourcing all that data, but using "select"
>>>> statements i only select partial data,
>>>> does the metapost limit apply to the whole dataset or just the scraps
>>>> covered by the select command?
>>>>
>>>>
>>>>
>>>> Am 2017-11-28 22:19, schrieb Benedikt Hallinger via Therion:
>>>>
>>>>>
>>>>> Hello Martin,
>>>>> is the blow limit of 4096 scraps still valid in the current version?
>>>>> Or is it already fixed so we can use more scraps?
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Dec 1, 2009 at 5:26 PM, Carl Magnuson <magnu213 at umn.edu>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> It looks like the solution is to issue the following metapost
>>>>>>> command:
>>>>>>> warningcheck := 0;
>>>>>>>
>>>>>>
>>>>>>
>>>>>> Indeed. The new limit will be 32768 and could not be increased further
>>>>>> in Metapost itself.
>>>>>>
>>>>>> The solution would be modification of how therion manages metapost
>>>>>> pictures (currently they are stored in files data.1 to data.4000, with
>>>>>> files data.4001 to data.4095 reserved for pattern definitions). This
>>>>>> numbering scheme could be modified to allow more file name prefixes
>>>>>> and consequently theoretically unlimited number of scraps processed by
>>>>>> metapost.
>>>>>>
>>>>>> On the other hand there is still pdfTeX limit which would not allow
>>>>>> much more scraps. PdfTeX uses internal registers for scraps
>>>>>> referencing (scrap data is included only once in pdf file and can be
>>>>>> referenced on multiple pages). You could avoid pdftex limit by using
>>>>>> SVG output (if SVG viewers would process large number of internal
>>>>>> references).
>>>>>>
>>>>>> In the longer-term future (a few years) I would like to use metapost
>>>>>> as a library instead of external metapost executable, which would
>>>>>> solve the problems with temporary files (and other problems as well).
>>>>>>
>>>>>> However adding it in a
>>>>>>> code metapost
>>>>>>> warningcheck := 0;
>>>>>>> endcode
>>>>>>> block seems to have no effect, mpost still fails on more then 4096
>>>>>>> scraps.
>>>>>>>
>>>>>>
>>>>>>
>>>>>> Therion currently inserts warningcheck:=1; before scraps without good
>>>>>> reason, so it will be fixed soon.
>>>>>>
>>>>>> If the new warningcheck setting would work for you, I would prefer not
>>>>>> to modify current file numbering scheme for metapost pictures and have
>>>>>> it fixed later with implementation of metapost library.
>>>>>>
>>>>>> Martin
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Therion mailing list
>>>>> Therion at speleo.sk
>>>>> https://mailman.speleo.sk/listinfo/therion
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Therion mailing list
>>>> Therion at speleo.sk
>>>> https://mailman.speleo.sk/listinfo/therion
>>>>
>>> _______________________________________________
>>> Therion mailing list
>>> Therion at speleo.sk
>>> https://mailman.speleo.sk/listinfo/therion
>>>
>>
> _______________________________________________
> Therion mailing list
> Therion at speleo.sk
> https://mailman.speleo.sk/listinfo/therion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.speleo.sk/pipermail/therion/attachments/20171229/24b3efc4/attachment.htm>
More information about the Therion
mailing list