<div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif">Hello, I have not investigated to details a lot, but the problem with this huge file is search for automatic joins. All the scraps are in one giant file and have a lot of common points so it tries to auto-connect every scrap with every scrap, therefore n^2 memory usage. Situation that should not happen in reality.</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">If you apply attached patch and disable automatic scrap joins - therion runs smoothly even with such a giant structure. But more proper solution would be to shift every scrap, that they will not have common points.</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">S.</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">diff --git a/thdb2d.cxx b/thdb2d.cxx</div><div class="gmail_default" style="font-family:tahoma,sans-serif">index b8c9720..7d7b72e 100644</div><div class="gmail_default" style="font-family:tahoma,sans-serif">--- a/thdb2d.cxx</div><div class="gmail_default" style="font-family:tahoma,sans-serif">+++ b/thdb2d.cxx</div><div class="gmail_default" style="font-family:tahoma,sans-serif">@@ -2428,6 +2428,7 @@ void thdb2d::pp_process_joins(<wbr>thdb2dprj * prj)</div><div class="gmail_default" style="font-family:tahoma,sans-serif"> </div><div class="gmail_default" style="font-family:tahoma,sans-serif"> // prida na koniec automaticke joiny</div><div class="gmail_default" style="font-family:tahoma,sans-serif"> jci = joincandlist.begin();</div><div class="gmail_default" style="font-family:tahoma,sans-serif">+ jci = joincandlist.end();</div><div class="gmail_default" style="font-family:tahoma,sans-serif"> while (jci != joincandlist.end()) {</div><div class="gmail_default" style="font-family:tahoma,sans-serif"> njci = jci;</div><div class="gmail_default" style="font-family:tahoma,sans-serif"> njci++;</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"></div><div class="gmail_extra"><br><div class="gmail_quote">On 27 December 2017 at 13:26, Benedikt Hallinger via Therion <span dir="ltr"><<a href="mailto:therion@speleo.sk" target="_blank">therion@speleo.sk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,<br>
for further testing i wrote a small script generating a sample cave of variable length.<br>
<br>
The bash-script generateTestCave.sh generates a th-file one main survey and *n* linked subusrveys with included scrap and is called as following:<br>
./generateTestCave.sh <number-of-surveys> <a href="http://test.th" rel="noreferrer" target="_blank">test.th</a><br>
eg "./generateTestCave.sh 1000 <a href="http://test.th" rel="noreferrer" target="_blank">test.th</a>" generates a survey "test" containing 1000 linked subsurveys containing each a suitable inline-copy of the <a href="http://scrap.th" rel="noreferrer" target="_blank">scrap.th</a>. On the main "test"-survey level also for each subsurvey a "map n-Plan" is generated.<br>
This is then selected in the "thconfig" file.<br>
<br>
With the contained thconig file one can pay a little to explore behavior.<br>
<br>
What i encounter is the following unexepected:<br>
With raising survey count the memory needed and time to compile increases very heavily. This is DESPITE i only select 3 distinct maps for output in lox and pdf plan projection.<br>
Why is that so? i expected, that the selection influences the scraps selected and thus compile time should nearly remain constant (at least only increase slightly due to more data processed and discarded)?<br>
Also, in the mailing list there was an encounter of strange behavior of the select/unselect command - do we have a bug here somewhere that also causes the behavior i see here?<div class="m_-6020714946283446200HOEnZb"><div class="m_-6020714946283446200h5"><br>
<br>
<br>
<br>
<br>
Am 2017-12-03 10:41, schrieb Benedikt Hallinger:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Thank, how could i switch this?<br>
I also wrote a small testcase that segfaulted at about 3000 scraps.<br>
<br>
Am 2017-12-02 9:55, schrieb Martin Budaj via Therion:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br>
<br>
there is no change regarding the limits in Therion. If there is a real<br>
need, following could be done:<br>
<br>
- in the current version of MetaPost, it's possible to used "double"<br>
arithmetic just by specifying a command line option, which practically<br>
eliminates MetaPost limits<br>
<br>
- instead of pdfTeX we could use LuaTeX to produce the PDFs. This<br>
doubles the number of registers available from 32768 to 65536.<br>
Registers are needed to reference the fragments of all<br>
scraps/sections; you usually need up to 6 of them for a scrap. So you<br>
would get maybe 12000 instead of 6000 scraps in one output file. It<br>
would require some substantial work to support LuaTeX (there is e.g.<br>
completely different font handling compared to PdfTeX).<br>
<br>
And yes, the limit applies just to the data selected for export.<br>
<br>
Martin<br>
<br>
<br>
On Tue, Nov 28, 2017 at 9:44 PM, Benedikt Hallinger via Therion<br>
<<a href="mailto:therion@speleo.sk" target="_blank">therion@speleo.sk</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Maybe another question:<br>
Assume a large cave with thousands of scraps.<br>
When i make a thconfig file sourcing all that data, but using "select"<br>
statements i only select partial data,<br>
does the metapost limit apply to the whole dataset or just the scraps<br>
covered by the select command?<br>
<br>
<br>
<br>
Am 2017-11-28 22:19, schrieb Benedikt Hallinger via Therion:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Hello Martin,<br>
is the blow limit of 4096 scraps still valid in the current version?<br>
Or is it already fixed so we can use more scraps?<br>
<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Tue, Dec 1, 2009 at 5:26 PM, Carl Magnuson <magnu213 at <a href="http://umn.edu" rel="noreferrer" target="_blank">umn.edu</a>><br>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
It looks like the solution is to issue the following metapost command:<br>
warningcheck := 0;<br>
</blockquote>
<br>
<br>
Indeed. The new limit will be 32768 and could not be increased further<br>
in Metapost itself.<br>
<br>
The solution would be modification of how therion manages metapost<br>
pictures (currently they are stored in files data.1 to data.4000, with<br>
files data.4001 to data.4095 reserved for pattern definitions). This<br>
numbering scheme could be modified to allow more file name prefixes<br>
and consequently theoretically unlimited number of scraps processed by<br>
metapost.<br>
<br>
On the other hand there is still pdfTeX limit which would not allow<br>
much more scraps. PdfTeX uses internal registers for scraps<br>
referencing (scrap data is included only once in pdf file and can be<br>
referenced on multiple pages). You could avoid pdftex limit by using<br>
SVG output (if SVG viewers would process large number of internal<br>
references).<br>
<br>
In the longer-term future (a few years) I would like to use metapost<br>
as a library instead of external metapost executable, which would<br>
solve the problems with temporary files (and other problems as well).<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
However adding it in a<br>
code metapost<br>
warningcheck := 0;<br>
endcode<br>
block seems to have no effect, mpost still fails on more then 4096<br>
scraps.<br>
</blockquote>
<br>
<br>
Therion currently inserts warningcheck:=1; before scraps without good<br>
reason, so it will be fixed soon.<br>
<br>
If the new warningcheck setting would work for you, I would prefer not<br>
to modify current file numbering scheme for metapost pictures and have<br>
it fixed later with implementation of metapost library.<br>
<br>
Martin<br>
</blockquote>
<br>
______________________________<wbr>_________________<br>
Therion mailing list<br>
<a href="mailto:Therion@speleo.sk" target="_blank">Therion@speleo.sk</a><br>
<a href="https://mailman.speleo.sk/listinfo/therion" rel="noreferrer" target="_blank">https://mailman.speleo.sk/list<wbr>info/therion</a><br>
</blockquote>
<br>
<br>
______________________________<wbr>_________________<br>
Therion mailing list<br>
<a href="mailto:Therion@speleo.sk" target="_blank">Therion@speleo.sk</a><br>
<a href="https://mailman.speleo.sk/listinfo/therion" rel="noreferrer" target="_blank">https://mailman.speleo.sk/list<wbr>info/therion</a><br>
</blockquote>
______________________________<wbr>_________________<br>
Therion mailing list<br>
<a href="mailto:Therion@speleo.sk" target="_blank">Therion@speleo.sk</a><br>
<a href="https://mailman.speleo.sk/listinfo/therion" rel="noreferrer" target="_blank">https://mailman.speleo.sk/list<wbr>info/therion</a><br>
</blockquote></blockquote>
</div></div><br>______________________________<wbr>_________________<br>
Therion mailing list<br>
<a href="mailto:Therion@speleo.sk" target="_blank">Therion@speleo.sk</a><br>
<a href="https://mailman.speleo.sk/listinfo/therion" rel="noreferrer" target="_blank">https://mailman.speleo.sk/list<wbr>info/therion</a><br>
<br></blockquote></div><br></div></div>