Fixing a point using standard deviation in different centrelines
Hi folks, Our survey has a location fix. The location has been taken 4 times, on 2 separate dates, with separate teams and equipment. As a result, I would like to have the fix data in 2 centreline sections, each with its own date, team and instrument. Each fix has its standard deviation values set, so that Therion can calculate the weighted position. When I do that, Therion miscalculates the position. If I put the fix statements all in the same centreline section, Therion calculates the position correctly. It is definitely using all four "fix" statements, as the positions are different if I remove the second centreline block completely. But when they are split over separate centreline blocks, it seems to reduce the weighting that it applies to the fixes in the second block. The specified standard deviations don't seem to affect the bug. I have attached a demo that shows the problem, along with .3d and .kml outputs that show the incorrect positions. Am I doing something wrong, or is this a Therion bug? Cheers, Tarquin
Freaky. I have done this a number of times, but of course never looked critically at the results. Too trusting. So I can replicate the effect for both the scenarios you describe using my usual 'loop-closure survex' in therion.ini. If I switch to 'loop-closure therion' then both scenarios give co-incident stations 1a and 1b, which I believe is what you are after. So all these years I've been using the 'better' survex loop closure to avoid one bug, and inadvertently falling for another! Bruce -----Original Message----- From: Therion <therion-bounces@speleo.sk> On Behalf Of Tarquin Wilton-Jones via Therion Sent: Wednesday, 16 September 2020 23:48 To: Therion <therion@speleo.sk> Cc: Tarquin Wilton-Jones <tarquin.wilton-jones@ntlworld.com> Subject: [Therion] Fixing a point using standard deviation in different centrelines Hi folks, Our survey has a location fix. The location has been taken 4 times, on 2 separate dates, with separate teams and equipment. As a result, I would like to have the fix data in 2 centreline sections, each with its own date, team and instrument. Each fix has its standard deviation values set, so that Therion can calculate the weighted position. When I do that, Therion miscalculates the position. If I put the fix statements all in the same centreline section, Therion calculates the position correctly. It is definitely using all four "fix" statements, as the positions are different if I remove the second centreline block completely. But when they are split over separate centreline blocks, it seems to reduce the weighting that it applies to the fixes in the second block. The specified standard deviations don't seem to affect the bug. I have attached a demo that shows the problem, along with .3d and .kml outputs that show the incorrect positions. Am I doing something wrong, or is this a Therion bug? Cheers, Tarquin
So I can replicate the effect for both the scenarios you describe using my usual 'loop-closure survex' in therion.ini. If I switch to 'loop-closure therion' then both scenarios give co-incident stations 1a and 1b, which I believe is what you are after.
Oh! That's weird. Survex calculates it correctly when I use it in a Survex file with separate begin/end sections, so I don't think the bug is there (but maybe Therion puts it into a certain structure that confuses Survex). Perhaps Olli or Wookey might have some insight into why Therion gets it wrong when using Survex for loop closure...
So all these years I've been using the 'better' survex loop closure to avoid one bug, and inadvertently falling for another!
It's a shame the switch is in the ini configuration, rather than something I can control in the survey or centreline level. Then I could fix this bug without having to resort to using Therion loop closure elsewhere. And unfortunately, it looks like Therion loop closure bugs are likely to hit the survey because of how I am (correctly) using standard deviation on both the fixed stations and legs between them. Don't suppose you can magic up some other workaround to the issue, or should I pester people and hope someone can fix it? I will put it on the issues list tomorrow anyway, just in case (unless Olli or Wookey jump in and say it is a Survex bug).
So I can replicate the effect for both the scenarios you describe using my usual 'loop-closure survex' in therion.ini. If I switch to 'loop-closure therion' then both scenarios give co-incident stations 1a and 1b, which I believe is what you are after.
Oh! That's weird. Survex calculates it correctly when I use it in a Survex file with separate begin/end sections, so I don't think the bug is there (but maybe Therion puts it into a certain structure that confuses Survex).
Perhaps Olli or Wookey might have some insight into why Therion gets it wrong when using Survex for loop closure...
I have worked out (I think) what Therion is doing wrong. Only the values in the last centreline section get used for the fix of a point that appears in multiple centreline sections. So in my demonstration, it uses the last three, and not the first one, for fixing point 1b. This could be because when using Survex, it is processing the centreline blocks in reverse order, and either issuing a "*solve" command between them, or just taking the result as a non-modifiable result (so it ignores it when the next centreline block tries to modify it). Eg. this could happen if it is considering each centreline block individually instead of all at once. That sounds like a recipe for hidden faults like this, so I do hope it is not something fundamental in the way Therion handles things. I also wonder how many other things will be secretly ignored. https://github.com/therion/therion/issues/269
Hi folks, Stacho fixed these "when using Survex loop closure" bugs in yesterday's development release: https://github.com/therion/therion/issues/269 Therion miscalculates fixes in multiple centreline blocks when using Survex loop closure https://github.com/therion/therion/issues/270 Therion cannot have exactly 2 fixes for a station when using Survex loop closure And this Survex compatibility bug: https://github.com/therion/therion/issues/263 Scale value should default to 1 when omitted from "calibrate" commands I have checked and verified the fixes, and my surveys still look right. But could someone with a bigger dataset check that it didn't break anything else when using Survex loop closures. That particular fix looks like the kind that could create regressions. It only improved things for me though :) Yes, your cave entrance locations will change if you ever ran into #269 without realising - they are supposed to. And yes, you might find some parts of the cave change size if you ever hit the calibrate statement bug. Cheers, Tarquin
I tried to compile the model of the Hirlatz cave with it and it looks good, from a big picture view. The passages shifted sometimes slightly, as expected. When trying to compile a specific plan map view PDF i get an error about UTF8. The compile runs fine with release-therion 5.5.1, however! (i don't know if this is related) therion 5.5.2+4683494 (2020-10-13) cavern - Survex 1.2.42 initialization file: therion.ini reading ... done This is pdfTeX, Version 3.14159265-2.6-1.40.21 (TeX Live 2020/Debian) (preloaded format=pdftex) restricted \write18 enabled. entering extended mode (./fonttest.tex checking optional fonts csr10 csti10 csbx10 csss10 csssi10 ... NOT INSTALLED This is pdfTeX, Version 3.14159265-2.6-1.40.21 (TeX Live 2020/Debian) (preloaded format=pdftex) restricted \write18 enabled. entering extended mode (./fonttest.tex checking optional fonts cmcyr10 cmcti10 cmcbx10 cmcss10 cmcssi10 ... NOT INSTALLED configuration file: ../../Dokumente/cave/HVHO/svn/Hirlatzhoehle/therion/plan.DOP.map.thconfig reading ... done reading source files ... done preprocessing database ... done ./therion: warning -- year 1951 magnetic declination used for undated surveys undated surveys: 155.Wilder_Westen.Hirlatzhoehle 323.Westen.Hirlatzhoehle 326.Hochdonnerbach.Hirlatzhoehle scanning centreline tree ... done searching for centerline loops ... done calculating station coordinates ... done average loop error: 1.98% processing survey data ... done calculating basic statistics ... done processing extended elevation ... done processing references ... done selecting export objects ... done processing projection plan ... done average distortion: 3.21% writing ../../Dokumente/cave/HVHO/svn/Hirlatzhoehle/therion/../Hirlatz.plan.DOP.map.pdf ... ./therion: error -- Invalid utf-8 string! Am 2020-10-14 10:02, schrieb Tarquin Wilton-Jones via Therion:
Hi folks,
Stacho fixed these "when using Survex loop closure" bugs in yesterday's development release:
https://github.com/therion/therion/issues/269 Therion miscalculates fixes in multiple centreline blocks when using Survex loop closure
https://github.com/therion/therion/issues/270 Therion cannot have exactly 2 fixes for a station when using Survex loop closure
And this Survex compatibility bug:
https://github.com/therion/therion/issues/263 Scale value should default to 1 when omitted from "calibrate" commands
I have checked and verified the fixes, and my surveys still look right. But could someone with a bigger dataset check that it didn't break anything else when using Survex loop closures. That particular fix looks like the kind that could create regressions. It only improved things for me though :)
Yes, your cave entrance locations will change if you ever ran into #269 without realising - they are supposed to. And yes, you might find some parts of the cave change size if you ever hit the calibrate statement bug.
Cheers,
Tarquin _______________________________________________ Therion mailing list Therion@speleo.sk https://mailman.speleo.sk/listinfo/therion
I experimented a little bit more. In the thconfig file I set some header data. statistics explo off statistics topo off statistics carto off statistics copyright all If i comment out the line #statistics carto off the compilation works again (then showing the people of course). The same is true with "statistics carto all". Why does turning off the carto yield invalid UTF8 data? Am 2020-10-14 10:38, schrieb Benedikt Hallinger:
I tried to compile the model of the Hirlatz cave with it and it looks good, from a big picture view. The passages shifted sometimes slightly, as expected.
When trying to compile a specific plan map view PDF i get an error about UTF8. The compile runs fine with release-therion 5.5.1, however! (i don't know if this is related)
therion 5.5.2+4683494 (2020-10-13) cavern - Survex 1.2.42 initialization file: therion.ini reading ... done This is pdfTeX, Version 3.14159265-2.6-1.40.21 (TeX Live 2020/Debian) (preloaded format=pdftex) restricted \write18 enabled. entering extended mode (./fonttest.tex checking optional fonts csr10 csti10 csbx10 csss10 csssi10 ... NOT INSTALLED This is pdfTeX, Version 3.14159265-2.6-1.40.21 (TeX Live 2020/Debian) (preloaded format=pdftex) restricted \write18 enabled. entering extended mode (./fonttest.tex checking optional fonts cmcyr10 cmcti10 cmcbx10 cmcss10 cmcssi10 ... NOT INSTALLED configuration file: ../../Dokumente/cave/HVHO/svn/Hirlatzhoehle/therion/plan.DOP.map.thconfig reading ... done reading source files ... done preprocessing database ... done ./therion: warning -- year 1951 magnetic declination used for undated surveys undated surveys: 155.Wilder_Westen.Hirlatzhoehle 323.Westen.Hirlatzhoehle 326.Hochdonnerbach.Hirlatzhoehle scanning centreline tree ... done searching for centerline loops ... done calculating station coordinates ... done average loop error: 1.98% processing survey data ... done calculating basic statistics ... done processing extended elevation ... done processing references ... done selecting export objects ... done processing projection plan ... done average distortion: 3.21% writing ../../Dokumente/cave/HVHO/svn/Hirlatzhoehle/therion/../Hirlatz.plan.DOP.map.pdf ... ./therion: error -- Invalid utf-8 string!
Am 2020-10-14 10:02, schrieb Tarquin Wilton-Jones via Therion:
Hi folks,
Stacho fixed these "when using Survex loop closure" bugs in yesterday's development release:
https://github.com/therion/therion/issues/269 Therion miscalculates fixes in multiple centreline blocks when using Survex loop closure
https://github.com/therion/therion/issues/270 Therion cannot have exactly 2 fixes for a station when using Survex loop closure
And this Survex compatibility bug:
https://github.com/therion/therion/issues/263 Scale value should default to 1 when omitted from "calibrate" commands
I have checked and verified the fixes, and my surveys still look right. But could someone with a bigger dataset check that it didn't break anything else when using Survex loop closures. That particular fix looks like the kind that could create regressions. It only improved things for me though :)
Yes, your cave entrance locations will change if you ever ran into #269 without realising - they are supposed to. And yes, you might find some parts of the cave change size if you ever hit the calibrate statement bug.
Cheers,
Tarquin _______________________________________________ Therion mailing list Therion@speleo.sk https://mailman.speleo.sk/listinfo/therion
Therion mailing list Therion@speleo.sk https://mailman.speleo.sk/listinfo/therion
On Wed, Oct 14, 2020 at 10:38 AM Benedikt Hallinger <beni@hallinger.org> wrote:
When trying to compile a specific plan map view PDF i get an error about UTF8. The compile runs fine with release-therion 5.5.1, however! (i don't know if this is related)
Hi, there was no change in the processing of utf-8 strings between 5.5.1 and 5.5.2. Could you check the same data set is used by those versions? You should get this error if you use a unicode character outside of the BMP (see https://en.wikipedia.org/wiki/Plane_(Unicode)#Basic_Multilingual_Plane). Do you such exotic characters? Sending a minimal data sample would be helpful. M.
Hey Martin, thanks for responding. The exact same dataset compiles well with 5.5.1. As far as i know, no special characters are used here. As its rather huge, and i have no idea how to reduce the problem. It surely must be somewhere in the .th2 files "author", since that is probably the only place where this gets gathered from, isn't it? I enhanced the thtexfonts.cxx on line 247 to print out the offending string (if you want i make a PR for that) that now shows something looking like garbage: ./therion: error -- Invalid utf-8 string! (offending string: 'P]���U') I probably need to wade trough the data manually. Am 2020-10-14 20:04, schrieb Martin Budaj:
On Wed, Oct 14, 2020 at 10:38 AM Benedikt Hallinger <beni@hallinger.org> wrote:
When trying to compile a specific plan map view PDF i get an error about UTF8. The compile runs fine with release-therion 5.5.1, however! (i don't know if this is related)
Hi, there was no change in the processing of utf-8 strings between 5.5.1 and 5.5.2. Could you check the same data set is used by those versions?
You should get this error if you use a unicode character outside of the BMP (see https://en.wikipedia.org/wiki/Plane_(Unicode)#Basic_Multilingual_Plane). Do you such exotic characters?
Sending a minimal data sample would be helpful.
M. _______________________________________________ Therion mailing list Therion@speleo.sk https://mailman.speleo.sk/listinfo/therion
I probably need to wade trough the data manually.
QA folks start grinning. Generic instructions below: Take a copy of your dataset. Reproduce the problem. Comment out any joins. Recompile. Delete half the "input" statements, and their "equate". Recompile. Repeat until the problem disappears. If you get all of it gone and the problem is still happening, great. Repeat on any remaining sub-surveys, until the only files left are the ones needed to show the bug. Remove any contents of the file (maps etc.) bit by bit, until the problem disappears. Or send it to me, and I will do it.
What about multifile search for”P]” in text files? It is quite strange combination of characters and they are part of ASCII set. Martin Odesláno z iPhonu 14. 10. 2020 v 20:55, Tarquin Wilton-Jones via Therion <therion@speleo.sk>:
I probably need to wade trough the data manually.
QA folks start grinning. Generic instructions below:
Take a copy of your dataset. Reproduce the problem. Comment out any joins. Recompile. Delete half the "input" statements, and their "equate". Recompile. Repeat until the problem disappears. If you get all of it gone and the problem is still happening, great. Repeat on any remaining sub-surveys, until the only files left are the ones needed to show the bug. Remove any contents of the file (maps etc.) bit by bit, until the problem disappears.
Or send it to me, and I will do it. _______________________________________________ Therion mailing list Therion@speleo.sk https://mailman.speleo.sk/listinfo/therion
On 14/10/2020 20:08, Martin Sluka via Therion wrote:
What about multifile search for”P]” in text files? It is quite strange combination of characters and they are part of ASCII set.
If you have an editor that is working in the same character set as Therion, that could work. Definitely worth a try. I suspect the fault relates to character set conversion though. Such as an editor working in 8859-1, and Therion trying to interpret it in UTF-8. Most basic text editors don't let you control character set, and just work in the system default (Windows-1252 on English-based Windows systems). Even if they do, most users are not aware of what it all means, and just stick with the default.
I use BBEdit on MacOSX and it has all features you mentioned. Martin
14. 10. 2020 v 21:17, Tarquin Wilton-Jones via Therion <therion@speleo.sk>:
On 14/10/2020 20:08, Martin Sluka via Therion wrote:
What about multifile search for”P]” in text files? It is quite strange combination of characters and they are part of ASCII set.
If you have an editor that is working in the same character set as Therion, that could work. Definitely worth a try.
I suspect the fault relates to character set conversion though. Such as an editor working in 8859-1, and Therion trying to interpret it in UTF-8. Most basic text editors don't let you control character set, and just work in the system default (Windows-1252 on English-based Windows systems). Even if they do, most users are not aware of what it all means, and just stick with the default. _______________________________________________ Therion mailing list Therion@speleo.sk https://mailman.speleo.sk/listinfo/therion
OK, now i got something. Guess what - one of about 5 compiles runs trough. This leads me to think that there is a race condition or something else somewhere, overwriting memory of the to-be-checked string. I'm pretty sure now that this is not a problem with the dataset per se, but some memory issue of the compiled therion program. The following pattern resulted in several runs with 5.5.2 (n=failed, y=ok): nnnynnynnynynynynnny Am 2020-10-14 21:53, schrieb Martin Sluka via Therion:
I use BBEdit on MacOSX and it has all features you mentioned.
Martin
14. 10. 2020 v 21:17, Tarquin Wilton-Jones via Therion <therion@speleo.sk>:
On 14/10/2020 20:08, Martin Sluka via Therion wrote:
What about multifile search for”P]” in text files? It is quite strange combination of characters and they are part of ASCII set.
If you have an editor that is working in the same character set as Therion, that could work. Definitely worth a try.
I suspect the fault relates to character set conversion though. Such as an editor working in 8859-1, and Therion trying to interpret it in UTF-8. Most basic text editors don't let you control character set, and just work in the system default (Windows-1252 on English-based Windows systems). Even if they do, most users are not aware of what it all means, and just stick with the default. _______________________________________________ Therion mailing list Therion@speleo.sk https://mailman.speleo.sk/listinfo/therion
_______________________________________________ Therion mailing list Therion@speleo.sk https://mailman.speleo.sk/listinfo/therion
On 2020-10-14 22:05 +0200, Benedikt Hallinger wrote:
OK, now i got something.
Guess what - one of about 5 compiles runs trough. This leads me to think that there is a race condition or something else somewhere, overwriting memory of the to-be-checked string. I'm pretty sure now that this is not a problem with the dataset per se, but some memory issue of the compiled therion program.
The following pattern resulted in several runs with 5.5.2 (n=failed, y=ok): nnnynnynnynynynynnny
Oh dear. This could be fun to track down.... Better check 5.5.1 always works, not just more often. But if it does then careful examination of what changed is in order. Maybe send some others the dataset to see who can reproduce (windows/linux/macOS)? Wookey -- Principal hats: Linaro, Debian, Wookware, ARM http://wookware.org/
Hi there, i performed a git bisect (the very first in my lifetime, and that is really a nice and easy tool to search such things!!!). The commits at and before c7d41d9 compiled the cave sucessfully 10 of 10 times. 1c1fcf14e4eaf6e0bb89da513702ad9b89204ed8 is the first bad commit commit 1c1fcf14e4eaf6e0bb89da513702ad9b89204ed8 Author: mbudaj <m.budaj@gmail.com> Date: Wed Aug 19 07:25:53 2020 +0200 dynamically load EPSG/ESRI labels in Proj>=6 https://github.com/therion/therion/commit/1c1fcf14e4eaf6e0bb89da513702ad9b89... This compiles unstable like described before. Am 2020-10-14 22:32, schrieb Wookey:
On 2020-10-14 22:05 +0200, Benedikt Hallinger wrote:
OK, now i got something.
Guess what - one of about 5 compiles runs trough. This leads me to think that there is a race condition or something else somewhere, overwriting memory of the to-be-checked string. I'm pretty sure now that this is not a problem with the dataset per se, but some memory issue of the compiled therion program.
The following pattern resulted in several runs with 5.5.2 (n=failed, y=ok): nnnynnynnynynynynnny
Oh dear. This could be fun to track down....
Better check 5.5.1 always works, not just more often. But if it does then careful examination of what changed is in order. Maybe send some others the dataset to see who can reproduce (windows/linux/macOS)?
Wookey _______________________________________________ Therion mailing list Therion@speleo.sk https://mailman.speleo.sk/listinfo/therion
I just made a new issue ticket: https://github.com/therion/therion/issues/278 Am 2020-10-14 22:53, schrieb Benedikt Hallinger:
Hi there, i performed a git bisect (the very first in my lifetime, and that is really a nice and easy tool to search such things!!!).
The commits at and before c7d41d9 compiled the cave sucessfully 10 of 10 times.
1c1fcf14e4eaf6e0bb89da513702ad9b89204ed8 is the first bad commit commit 1c1fcf14e4eaf6e0bb89da513702ad9b89204ed8 Author: mbudaj <m.budaj@gmail.com> Date: Wed Aug 19 07:25:53 2020 +0200
dynamically load EPSG/ESRI labels in Proj>=6
https://github.com/therion/therion/commit/1c1fcf14e4eaf6e0bb89da513702ad9b89...
This compiles unstable like described before.
Am 2020-10-14 22:32, schrieb Wookey:
On 2020-10-14 22:05 +0200, Benedikt Hallinger wrote:
OK, now i got something.
Guess what - one of about 5 compiles runs trough. This leads me to think that there is a race condition or something else somewhere, overwriting memory of the to-be-checked string. I'm pretty sure now that this is not a problem with the dataset per se, but some memory issue of the compiled therion program.
The following pattern resulted in several runs with 5.5.2 (n=failed, y=ok): nnnynnynnynynynynnny
Oh dear. This could be fun to track down....
Better check 5.5.1 always works, not just more often. But if it does then careful examination of what changed is in order. Maybe send some others the dataset to see who can reproduce (windows/linux/macOS)?
Wookey _______________________________________________ Therion mailing list Therion@speleo.sk https://mailman.speleo.sk/listinfo/therion
Therion mailing list Therion@speleo.sk https://mailman.speleo.sk/listinfo/therion
Hi, I think that the problem is caused by this line: rv.prjname = thcs_get_label(params).c_str(); because rv.prjname now contains a pointer to a temporary (and already destroyed) string. st 14. 10. 2020 v 23:04 odesílatel Benedikt Hallinger <beni@hallinger.org> napsal:
I just made a new issue ticket: https://github.com/therion/therion/issues/278
Am 2020-10-14 22:53, schrieb Benedikt Hallinger:
Hi there, i performed a git bisect (the very first in my lifetime, and that is really a nice and easy tool to search such things!!!).
The commits at and before c7d41d9 compiled the cave sucessfully 10 of 10 times.
1c1fcf14e4eaf6e0bb89da513702ad9b89204ed8 is the first bad commit commit 1c1fcf14e4eaf6e0bb89da513702ad9b89204ed8 Author: mbudaj <m.budaj@gmail.com> Date: Wed Aug 19 07:25:53 2020 +0200
dynamically load EPSG/ESRI labels in Proj>=6
https://github.com/therion/therion/commit/1c1fcf14e4eaf6e0bb89da513702ad9b89...
This compiles unstable like described before.
Am 2020-10-14 22:32, schrieb Wookey:
On 2020-10-14 22:05 +0200, Benedikt Hallinger wrote:
OK, now i got something.
Guess what - one of about 5 compiles runs trough. This leads me to think that there is a race condition or something else somewhere, overwriting memory of the to-be-checked string. I'm pretty sure now that this is not a problem with the dataset per se, but some memory issue of the compiled therion program.
The following pattern resulted in several runs with 5.5.2 (n=failed, y=ok): nnnynnynnynynynynnny
Oh dear. This could be fun to track down....
Better check 5.5.1 always works, not just more often. But if it does then careful examination of what changed is in order. Maybe send some others the dataset to see who can reproduce (windows/linux/macOS)?
Wookey _______________________________________________ Therion mailing list Therion@speleo.sk https://mailman.speleo.sk/listinfo/therion
Therion mailing list Therion@speleo.sk https://mailman.speleo.sk/listinfo/therion
Therion mailing list Therion@speleo.sk https://mailman.speleo.sk/listinfo/therion
I think that the problem is caused by this line:
rv.prjname = thcs_get_label(params).c_str();
because rv.prjname now contains a pointer to a temporary (and already destroyed) string.
Bravo on the speedy bug hunting!
With every run the string changes
Heart sank as soon as I read that. Never as easy to narrow those down. Have to set up enough state beforehand to get the nonsense, so cutting the lines out of the file often makes the bug go away. I assume one of your cartographers was listed with an unexpected name on the PDF when it "worked". Or perhaps some other little string. Bisect is definitely the easiest way to do it. At least enough to give the dev an idea of what code to analyse.
Hello all, thanks for all the hints.
What about multifile search for”P]” in text files? It is quite strange combination of characters and they are part of ASCII set. I had that in mind, but i don't think its the problem here. If it would, 5.5.1 should fail too...
What about multifile search for”P]” With every run the string changes. I have seen that before with C++ code, that was when i tried to put out a memory adress instead of the actual string. Maybe the passed string is already memory garbage at this point?
QA folks start grinning. Generic instructions below: Uh, then i will have a grey beard finishing this. We talk about 130km of cave in way over 600 files here.
I also commented the the error line in the source code that bails out here, and everything (therion, but also the map) compiles just fine as it should. Also the cartographers names print nicely. Nonewithstanding i had a idea i went trough. A part of the cave is done using an inkscape plugin, which converts the svg data into th2 files. That usually goes good, and did to therion 5.5.1. I also was able to isolate a scrap line causing the problem (excerpt attached). However, if i craft a very simple thconfig, it compiles fine; and the isolated line looks not suspicious at all for me! The problem occurs only if i try to render the whole cave (in a very big scale 1:5000); if i craft a very simple thconfig and source only the passage th file, it goes OK. Could it be that somewhere some string buffer runs over when gathering all the cartographers? or that something else messes stuff up memory-wise, because the dataset is so big already? I'm out of ideas. Am 2020-10-14 20:34, schrieb Benedikt Hallinger:
Hey Martin, thanks for responding. The exact same dataset compiles well with 5.5.1. As far as i know, no special characters are used here.
As its rather huge, and i have no idea how to reduce the problem. It surely must be somewhere in the .th2 files "author", since that is probably the only place where this gets gathered from, isn't it?
I enhanced the thtexfonts.cxx on line 247 to print out the offending string (if you want i make a PR for that) that now shows something looking like garbage: ./therion: error -- Invalid utf-8 string! (offending string: 'P]���U')
I probably need to wade trough the data manually.
Am 2020-10-14 20:04, schrieb Martin Budaj:
On Wed, Oct 14, 2020 at 10:38 AM Benedikt Hallinger <beni@hallinger.org> wrote:
When trying to compile a specific plan map view PDF i get an error about UTF8. The compile runs fine with release-therion 5.5.1, however! (i don't know if this is related)
Hi, there was no change in the processing of utf-8 strings between 5.5.1 and 5.5.2. Could you check the same data set is used by those versions?
You should get this error if you use a unicode character outside of the BMP (see https://en.wikipedia.org/wiki/Plane_(Unicode)#Basic_Multilingual_Plane). Do you such exotic characters?
Sending a minimal data sample would be helpful.
M. _______________________________________________ Therion mailing list Therion@speleo.sk https://mailman.speleo.sk/listinfo/therion
Therion mailing list Therion@speleo.sk https://mailman.speleo.sk/listinfo/therion
I sent Martin a dataset to investigate. Hopefully he can spot something... Am 2020-10-14 21:40, schrieb Benedikt Hallinger:
Hello all, thanks for all the hints.
What about multifile search for”P]” in text files? It is quite strange combination of characters and they are part of ASCII set. I had that in mind, but i don't think its the problem here. If it would, 5.5.1 should fail too...
What about multifile search for”P]” With every run the string changes. I have seen that before with C++ code, that was when i tried to put out a memory adress instead of the actual string. Maybe the passed string is already memory garbage at this point?
QA folks start grinning. Generic instructions below: Uh, then i will have a grey beard finishing this. We talk about 130km of cave in way over 600 files here.
I also commented the the error line in the source code that bails out here, and everything (therion, but also the map) compiles just fine as it should. Also the cartographers names print nicely.
Nonewithstanding i had a idea i went trough. A part of the cave is done using an inkscape plugin, which converts the svg data into th2 files. That usually goes good, and did to therion 5.5.1.
I also was able to isolate a scrap line causing the problem (excerpt attached). However, if i craft a very simple thconfig, it compiles fine; and the isolated line looks not suspicious at all for me!
The problem occurs only if i try to render the whole cave (in a very big scale 1:5000); if i craft a very simple thconfig and source only the passage th file, it goes OK. Could it be that somewhere some string buffer runs over when gathering all the cartographers? or that something else messes stuff up memory-wise, because the dataset is so big already?
I'm out of ideas.
Am 2020-10-14 20:34, schrieb Benedikt Hallinger:
Hey Martin, thanks for responding. The exact same dataset compiles well with 5.5.1. As far as i know, no special characters are used here.
As its rather huge, and i have no idea how to reduce the problem. It surely must be somewhere in the .th2 files "author", since that is probably the only place where this gets gathered from, isn't it?
I enhanced the thtexfonts.cxx on line 247 to print out the offending string (if you want i make a PR for that) that now shows something looking like garbage: ./therion: error -- Invalid utf-8 string! (offending string: 'P]���U')
I probably need to wade trough the data manually.
Am 2020-10-14 20:04, schrieb Martin Budaj:
On Wed, Oct 14, 2020 at 10:38 AM Benedikt Hallinger <beni@hallinger.org> wrote:
When trying to compile a specific plan map view PDF i get an error about UTF8. The compile runs fine with release-therion 5.5.1, however! (i don't know if this is related)
Hi, there was no change in the processing of utf-8 strings between 5.5.1 and 5.5.2. Could you check the same data set is used by those versions?
You should get this error if you use a unicode character outside of the BMP (see https://en.wikipedia.org/wiki/Plane_(Unicode)#Basic_Multilingual_Plane). Do you such exotic characters?
Sending a minimal data sample would be helpful.
M. _______________________________________________ Therion mailing list Therion@speleo.sk https://mailman.speleo.sk/listinfo/therion
Therion mailing list Therion@speleo.sk https://mailman.speleo.sk/listinfo/therion
Therion mailing list Therion@speleo.sk https://mailman.speleo.sk/listinfo/therion
participants (8)
-
Benedikt Hallinger -
Bruce Mutton -
Martin Budaj -
Martin Sluka -
Martin Sluka -
Matěj Plch -
Tarquin Wilton-Jones -
Wookey