survex to therion conversion script
A couple of people asked about this, so here it is under a sensible title so people can find it in the archives. Below is a perl script which does a reasonable job of converting .svx files to .th files. Written by Olly. I have noticed a few problems, which I might as well document here as as good a place as any: (I was going to tidy them up and send them to Ol as I was given said script for 'testing'). *calibrate declination comes out wrong. It should be: *calibrate declination n -> declination n degrees therion doesn't understand 'ignoreall' - needs commenting out Doesn't try to deal with different 'team' syntax - but could be fixed to get most of them right ((lower case them all), 'pics'->'pictures', 'disto'->'length #disto' etc) Only significant problem was with 'overview' files which include a number of others. * All the equates need enclosing in 'centreline'/'endcentreline' * The converter reads in any 'included' files and inserts them. It stopped after doing two of these and truncated the file. It should probably just convert '*include foo' to 'input foo.th' It once inserted a top-level 'dummy' survey where one wasn't needed. This is a feature of converting files individually rather than as a coherent dataset, I suspect. It could probably do a better job if it 'spidered' the dataset from a top-level overview file, but that's a lot more work :-) I found one case where the ';' remained for the comment char instead of it getting converted to '#'. I need to look at that or send ol the offending file. Then you can do this to convert a whole directory: for FILE in ls *.svx; do FILE=echo $FILE | sed "s/.svx//"; echo "Converting $FILE.svx"; svx2th $FILE.svx > $FILE.th; done #!/usr/bin/perl -w use strict; # svx2th v0.1 # Copyright (C) Olly Betts 2004 sub convert_file($); my $in_survey = 0; my $had_fix = 0; print "encoding iso8859-1\n"; for my $filename (@ARGV) { convert_file($filename); } if ($in_survey) { print "endsurvey dummy\n"; } sub convert_file($) { my $filename = shift; open F, "<", $filename or die "$filename: $!\n"; my @lines = <F>; close F; my $in_centre_line = 0; my $lineno = 0; my $dummy_survey = -1; foreach $_ (@lines) { ++$lineno; # Replace ; with # as comment separator. # FIXME won't cope with ; in a filename or *title s/;/#/; # Comment out "*export" and "*entrance" as there seems to be no # equivalent of either. if (/^\s*\*\s*(?:export|entrance)\b/i) { print "#$_"; next; } if ($in_centre_line) { if (/^\s*\*/ && !/^\s*\*\s*(?:date|calibrate|fix|equate|data|instrument|units|sd|infer|flags|team)\b/i) { print "endcentreline\n"; $in_centre_line = 0; } } else { if (/^\s*[^\s*#]/ || /^\s*\*\s*(?:date|calibrate|fix|equate|data|instrument|units|sd|infer|flags|team)\b/) { if (!$in_survey) { # Therion can't handle these outside a centreline which # must be inside a survey, so we have to add a dummy # top-level survey. print "survey dummy -title \"Therion is crap\"\n"; ++$in_survey; } print "centreline\n"; $in_centre_line = 1; } } # *begin <survey> -> survey <survey> if (s/^(\s*)\*(\s*)begin\b(\s*)(\S+)/$1$2survey$3$4 -title "$4" /i) { ++$in_survey; print $_; next; } # *begin -> <nothing> # The *begin will cause an endcentreline / centreline pair to be # output which hopefully prevents settings from escaping. However # this doesn't restore the old settings, just undoes any new ones. # FIXME: Need to address this somehow... if (/^(\s*)\*(\s*)begin\b[ \t]*$/i) { ++$in_survey; if ($dummy_survey != 0) { # FIXME: doesn't coped with nested *begin with no arguments... die "This convertor doesn't currently handle nested *begin with no arguments\n"; } $dummy_survey = $in_survey; print "#$_"; next; } # *end [<survey>] -> endsurvey [<survey>] if (s/^(\s*)\*(\s*)end\b/$1$2endsurvey/i) { if ($dummy_survey == $in_survey) { $_ = "#$_"; $dummy_survey = -1; } --$in_survey; if ($in_centre_line) { print "endcentreline\n"; $in_centre_line = 0; } print $_; next; } # *title <title> -> # -title <title> # FIXME: just comment out for now - should really convert to -title on # the "survey" line. if (s/^(\s*)\*\s*title\b\s*/$1# -title /i) { print $_; next; } # *team and *instrument format is unspecified in Survex, and they're # just informational so comment them out for now... # NB therion seems to be case sensitive so "Compass" isn't a valid role # ("compass" is)... # NB in *team pics -> pictures if (s/^(\s*)\*(\s*(?:team|instrument))\b/#$1$2/i) { print $_; next; } # *include -> literal text inclusion. # Note that the *include means an implicit *begin, but the output # may not reflect this correctly (since we can't handle *begin # with no survey name anyway...) if (/^\s*\*\s*include\s*"?([^"\s]*)/i) { # Use Unix path separators (/ not \) - Survex understands either on # either platform. my $filename = $1; $filename =~ s!\\!/!g; $filename .= '.svx' unless $filename =~ /\.svx$/i; convert_file($filename); next; } # survey.subsurvey.12 -> 12@subsurvey.survey if (s/^(\s*)\*(\s*equate)\b/$1$2/i) { # Ensure that a comment separator doesn't get eaten by station name. s/(\S)#/$1 #/; s/(\S+)\.(\S+)/"$2\@".join(".",reverse split m!\.!, $1)/ge; print $_; next; } if (/^\s*\*\s*fix\b/i) { $had_fix = 1; } s/^(\s*)\*/$1/; print; } } Wookey -- Aleph One Ltd, Bottisham, CAMBRIDGE, CB5 9BA, UK Tel +44 (0) 1223 811679 work: http://www.aleph1.co.uk/ play: http://www.chaos.org.uk/~wookey/
On Wed, Jan 12, 2005 at 02:54:53AM +0000, Wookey wrote:
It once inserted a top-level 'dummy' survey where one wasn't needed. This is a feature of converting files individually rather than as a coherent dataset, I suspect. It could probably do a better job if it 'spidered' the dataset from a top-level overview file, but that's a lot more work :-)
Um, the script is designed to crawl over an included tree of .svx files to produce a single .th file. So this is already done (and in fact more work would be required if you wanted a usable tree of .th files). I probably should have documented that, but to be honest the whole thing is barely more than a proof of concept - mostly it was quicker and a lot less tedious than converting all the data by hand, plus I could just rerun it if we found any typos in the survex dataset.
Then you can do this to convert a whole directory: for FILE in ls *.svx; do FILE=echo $FILE | sed "s/.svx//"; echo "Converting $FILE.svx"; svx2th $FILE.svx > $FILE.th; done
You seem to have lost some backticks in there: for FILE in *.svx; do FILE=`echo $FILE | sed "s/.svx//"`; echo "Converting $FILE.svx"; svx2th $FILE.svx > $FILE.th; done But the idea is that you just run it using: ./svx2th TOPLEVEL.svx > SINGLEFILE.th But I'm not sure a perl script is really a good approach. At least it stops you having to maintain two parallel versions of your data, but the two formats are sufficiently different that it's not just a matter of stripping out the "*"s and changing the comment characters, as I'd initially hoped. It's also hard to hide this machinery away from the user - for example, there's the danger that you change the generated .th file and lose the changes next time you run the script. I looked at bolting the .svx parsing code into Therion. If that could be done you'd just include a .svx file instead of a .th one and everything would just work. Unfortunately the therion code has a lot of Czech or Slovak variable names and comments and we had a survey to draw up for the BCRA conference, so I took the safer route and hacked up a bit of perl. I also wondered about being able to include a .3d file instead - that would probably actually be easier, as the .3d file reading code is designed as a separate module, whereas the .svx reading code isn't. And it would avoid the annoyance of having to choose at compile time whether you want to use Survex for loop closure or not. As it is the prebuilt Windows and debian packages for therion are useless to anyone who actually cares if their stations are in the same place in Survex and therion, which is rather sad... Cheers, Olly
I also wondered about being able to include a .3d file instead - that would probably actually be easier, as the .3d file reading code is designed as a separate module, whereas the .svx reading code isn't.
Great idea, Olly! This is probably ideal solution also for people, that are starting with therion and have their centerline data in different package. If this will be possible, they will not need to convert their dataset to therion format. And reading of .3d file (or any other with processed data, .plt for example) is usually trivial. So I suggest to add new command: input-centreline <filename> It will add survey stations and shots from given (.3d, later may be compass, toporobot, vtopo etc.) file to current survey. This should not be too much work. Any comments? Regards, S.
+++ Stacho Mudrak [05-01-12 09:49 +0100]:
So I suggest to add new command:
input-centreline <filename>
It will add survey stations and shots from given (.3d, later may be compass, toporobot, vtopo etc.) file to current survey. This should not be too much work.
Any comments?
_Now_ he tells me :-) Wookey -- Aleph One Ltd, Bottisham, CAMBRIDGE, CB5 9BA, UK Tel +44 (0) 1223 811679 work: http://www.aleph1.co.uk/ play: http://www.chaos.org.uk/~wookey/
On Wed, 12 Jan 2005, Wookey wrote:
+++ Stacho Mudrak [05-01-12 09:49 +0100]:
So I suggest to add new command:
input-centreline <filename>
It will add survey stations and shots from given (.3d, later may be compass, toporobot, vtopo etc.) file to current survey. This should not be too much work.
Any comments?
_Now_ he tells me :-)
This sounds very sensible and would nicely decouple the survex and therion files/data. -- Simeon
Stacho Mudrak wrote:
Great idea, Olly!
It certainly is. I've been wondering about the problems of keeping two datasets up to date. If the conversion from svx to .th requires any manual editing and is beyond scripting in a Makefile then keeping edits and fixes in sync would become a huge pain.
So I suggest to add new command:
input-centreline <filename>
It will add survey stations and shots from given (.3d, later may be compass, toporobot, vtopo etc.) file to current survey. This should not be too much work.
Any comments?
When you say 'current survey' I'm assuming that this would account for putting dotted station names into subsurveys. I'd need to process a whole survey hierarchy at once to benefit from the loop closing in cavern. Cheers, John
John Pybus wrote:
When you say 'current survey' I'm assuming that this would account for putting dotted station names into subsurveys. I'd need to process a whole survey hierarchy at once to benefit from the loop closing in cavern.
Well, this make sense only in the case, if user will create simmilar (up to some level) survey structure in therion, like it is in survex data he wants to input. So if therion will find station foo.1 and there will be corresponging subsurvey foo in therion data (at survey level, where input-centerline appears), it will put station 1 into this subsurvey. For other cases, we would like to add scrap option called -station-name-prefix/suffix, which will specify what characters to add to station names specified in scrap. It is usefull also for therion data, if scrap is referencing stations from some subsurvey. It will simplyfy entering station names a lot. In fact, survey hierarchy is not necessary, if you are drawing maps. There is a map hierarchy, which is independent on survey hierarchy, so you may define map structure like map floor1 f1scrap1 f1scrap2 endmap map floor2 f2scrap1 endmap map whole-cave floor2 floor1 endmap And select maps using their names in configuration file. In the future, there is also possibility to add some filter options to input-centerline command, that will import only stations with certain prefix (without this prefix) and you will be able to import same file in different survey. But I am not sure whether this is necessary. Regards, S.
On Wed, Jan 12, 2005 at 09:49:24AM +0100, Stacho Mudrak wrote:
This is probably ideal solution also for people, that are starting with therion and have their centerline data in different package. If this will be possible, they will not need to convert their dataset to therion format. And reading of .3d file (or any other with processed data, .plt for example) is usually trivial.
The hardest part of reading the file formats is pinning them down. Larry Fish was very helpful in resolving minor ambiguities in the Compass file format descriptions, which was very useful when writing the Survex support for Compass formats. Without a precise spec or a lot of sample data from a range of sources, you're likely to end up handling just a subset of the format. But generally it's going to be easier for processed data formats than unprocessed data formats as they're written by machines, so the format-as-defined-by-use has a lot less variation, even if you don't have a formal specification.
So I suggest to add new command:
input-centreline <filename>
It will add survey stations and shots from given (.3d, later may be compass, toporobot, vtopo etc.) file to current survey. This should not be too much work.
You get compass "for free" with the Survex "img" library - it transparently reads .plt files (and processed CMAP files). Cheers, Olly
participants (5)
-
John Pybus -
Olly Betts -
Simeon Warner -
Stacho Mudrak -
Wookey