Creating PDF/A map files
Hello everyone - I propose a new feature for Therion. This will probably take some work, and I am sure there will be discussion about how to implement it. It seems to me that the maps we produce with Therion are likely going to be stored for a very long time, perhaps running into multiple tens of years. As we all know, computer technology over that amount of time will change drastically. Just think about the contrast in both hardware and software in the last 25 years - from Windows 95 running on 486dx processors to Linux and Windows 10 running on i7 and i9 processors. I think we have some obligation to make sure the cave maps we generate are still usable many years from now. Saving them in PDF format is a large - but incomplete - step in that direction. The new feature I propose is to modify the PDF creation code so that it produces files that are PDF/A version 1b (or possibly version 2) compliant. https://en.wikipedia.org/wiki/PDF/A I have checked all of the PDF files I created in Therion, and none of them are flagged as PDF/A compliant. It is possible that they are, in fact, compliant and simply do not have the necessary flag. The experts can check that against the PDF/A specifications. Existing PDF documents can be checked for PDF/A compliance with a command-line tool called "verapdf". The web site for that tool is https://openpreservation.org/products/verapdf/ It is possible to use GhostScript to transform an existing PDF into a PDF/A file. The command line is daunting. https://www.mcbsys.com/blog/2018/10/batch-convert-pdf-to-pdf-a-2018-edition/ I tried the GhostScript conversion on one of my Therion maps. Immediately at startup it produced this message three times: "GPL Ghostscript 9.53.3: UTF16BE text string detected in DOCINFO cannot be represented in XMP for PDF/A1, reverting to normal PDF output" The process continued running and took about 10 minutes. The resulting file failed verapdf analysis. It also increased the file size from 4.3 megabytes to over 52 megabytes! The output file displayed correctly in Okular. I do not have any idea how Therion produces PDF files. It probably uses some combination of TeX and GhostScript to get it done. The new feature may be as simple as adding some additional parameters to the command lines that call the external programs. Let the discussion begin! :-) -- Bill Gee
Hi, instalation of veraPDF was straightforward. For a test run i get this: <details passedRules="96" failedRules="7" passedChecks="25551" failedChecks="1846"> This will be a long road. But your idea is quite good, as it will preserve valuable work. OTOH the PDFs are just results, what is really valuable would be the source files. And they are pretty good to archive already, as they are just text files! Therion should be fine to run a long time into the future, hopefully. And as its OpenSource, there are no constraints whatsoever to fix bugs in the future to make it work again... Am 2020-10-22 15:17, schrieb Bill Gee:
Hello everyone -
I propose a new feature for Therion. This will probably take some work, and I am sure there will be discussion about how to implement it.
It seems to me that the maps we produce with Therion are likely going to be stored for a very long time, perhaps running into multiple tens of years. As we all know, computer technology over that amount of time will change drastically. Just think about the contrast in both hardware and software in the last 25 years - from Windows 95 running on 486dx processors to Linux and Windows 10 running on i7 and i9 processors.
I think we have some obligation to make sure the cave maps we generate are still usable many years from now. Saving them in PDF format is a large - but incomplete - step in that direction.
The new feature I propose is to modify the PDF creation code so that it produces files that are PDF/A version 1b (or possibly version 2) compliant.
https://en.wikipedia.org/wiki/PDF/A [1]
I have checked all of the PDF files I created in Therion, and none of them are flagged as PDF/A compliant. It is possible that they are, in fact, compliant and simply do not have the necessary flag. The experts can check that against the PDF/A specifications.
Existing PDF documents can be checked for PDF/A compliance with a command-line tool called "verapdf". The web site for that tool is
https://openpreservation.org/products/verapdf/ [2]
It is possible to use GhostScript to transform an existing PDF into a PDF/A file. The command line is daunting.
https://www.mcbsys.com/blog/2018/10/batch-convert-pdf-to-pdf-a-2018-edition/ [3]
I tried the GhostScript conversion on one of my Therion maps. Immediately at startup it produced this message three times:
"GPL Ghostscript 9.53.3: UTF16BE text string detected in DOCINFO cannot be represented in XMP for PDF/A1, reverting to normal PDF output"
The process continued running and took about 10 minutes. The resulting file failed verapdf analysis. It also increased the file size from 4.3 megabytes to over 52 megabytes! The output file displayed correctly in Okular.
I do not have any idea how Therion produces PDF files. It probably uses some combination of TeX and GhostScript to get it done. The new feature may be as simple as adding some additional parameters to the command lines that call the external programs.
Let the discussion begin! :-)
--
Bill Gee
Links: ------ [1] https://en.wikipedia.org/wiki/PDF/A [2] https://openpreservation.org/products/verapdf/ [3] https://www.mcbsys.com/blog/2018/10/batch-convert-pdf-to-pdf-a-2018-edition/ _______________________________________________ Therion mailing list Therion@speleo.sk https://mailman.speleo.sk/listinfo/therion
Hi, This is a good idea, but I totally agree with Benedikt, the source files (both .th and .th2) are the most important files as this are text files that contains all the information of the survey and the drawing. For me, these are the files that should be archived and shared. Moreover, if I good understood, the pdf/A-1 does not allow transparency and layers. I am not a pdf specialist, but it seems to me that is not compatible with Therion outputs (for instance, If I open a pdf from Therion in Illustrator, I see sub-layers). Pdf/A-2 seems to be better as it allows both transparency and layers. Cheers, Xavier
Le 22 oct. 2020 à 15:43, Benedikt Hallinger <beni@hallinger.org> a écrit :
Hi, instalation of veraPDF was straightforward.
For a test run i get this: <details passedRules="96" failedRules="7" passedChecks="25551" failedChecks="1846"> This will be a long road.
But your idea is quite good, as it will preserve valuable work. OTOH the PDFs are just results, what is really valuable would be the source files. And they are pretty good to archive already, as they are just text files! Therion should be fine to run a long time into the future, hopefully. And as its OpenSource, there are no constraints whatsoever to fix bugs in the future to make it work again...
Am 2020-10-22 15:17, schrieb Bill Gee:
Hello everyone - I propose a new feature for Therion. This will probably take some work, and I am sure there will be discussion about how to implement it. It seems to me that the maps we produce with Therion are likely going to be stored for a very long time, perhaps running into multiple tens of years. As we all know, computer technology over that amount of time will change drastically. Just think about the contrast in both hardware and software in the last 25 years - from Windows 95 running on 486dx processors to Linux and Windows 10 running on i7 and i9 processors. I think we have some obligation to make sure the cave maps we generate are still usable many years from now. Saving them in PDF format is a large - but incomplete - step in that direction. The new feature I propose is to modify the PDF creation code so that it produces files that are PDF/A version 1b (or possibly version 2) compliant. https://en.wikipedia.org/wiki/PDF/A <https://en.wikipedia.org/wiki/PDF/A> [1] I have checked all of the PDF files I created in Therion, and none of them are flagged as PDF/A compliant. It is possible that they are, in fact, compliant and simply do not have the necessary flag. The experts can check that against the PDF/A specifications. Existing PDF documents can be checked for PDF/A compliance with a command-line tool called "verapdf". The web site for that tool is https://openpreservation.org/products/verapdf/ <https://openpreservation.org/products/verapdf/> [2] It is possible to use GhostScript to transform an existing PDF into a PDF/A file. The command line is daunting. https://www.mcbsys.com/blog/2018/10/batch-convert-pdf-to-pdf-a-2018-edition/ <https://www.mcbsys.com/blog/2018/10/batch-convert-pdf-to-pdf-a-2018-edition/> [3] I tried the GhostScript conversion on one of my Therion maps. Immediately at startup it produced this message three times: "GPL Ghostscript 9.53.3: UTF16BE text string detected in DOCINFO cannot be represented in XMP for PDF/A1, reverting to normal PDF output" The process continued running and took about 10 minutes. The resulting file failed verapdf analysis. It also increased the file size from 4.3 megabytes to over 52 megabytes! The output file displayed correctly in Okular. I do not have any idea how Therion produces PDF files. It probably uses some combination of TeX and GhostScript to get it done. The new feature may be as simple as adding some additional parameters to the command lines that call the external programs. Let the discussion begin! :-) -- Bill Gee Links: ------ [1] https://en.wikipedia.org/wiki/PDF/A <https://en.wikipedia.org/wiki/PDF/A> [2] https://openpreservation.org/products/verapdf/ <https://openpreservation.org/products/verapdf/> [3] https://www.mcbsys.com/blog/2018/10/batch-convert-pdf-to-pdf-a-2018-edition/ <https://www.mcbsys.com/blog/2018/10/batch-convert-pdf-to-pdf-a-2018-edition/> _______________________________________________ Therion mailing list Therion@speleo.sk <mailto:Therion@speleo.sk> https://mailman.speleo.sk/listinfo/therion <https://mailman.speleo.sk/listinfo/therion>
Therion mailing list Therion@speleo.sk <mailto:Therion@speleo.sk> https://mailman.speleo.sk/listinfo/therion <https://mailman.speleo.sk/listinfo/therion>
On 2020-10-22 08:17 -0500, Bill Gee wrote:
The new feature I propose is to modify the PDF creation code so that it produces files that are PDF/A version 1b (or possibly version 2) compliant. https://en.wikipedia.org/wiki/PDF/A
Therion would need to use PDF/A-2 because we need tranparency (I think). And whilst this is mostly sensible why on earth do they think banning LZW compression is a good idea? That's probably why your file expanded from 4 to 52MB. The last patents on it expired in 2004 and free implementations have been available for decades. If it wasn't for that we could have made it a standard output format (because self-contained files are a very good thing, and I'd expect my therion PDFs to already have this property). We can gzip the files after creation but a foo.PDF.gz is a lot less handy than a foo.PDF with internal compression. Wookey -- Principal hats: Linaro, Debian, Wookware, ARM http://wookware.org/
we need tranparency (I think)
Render your survey to SVG, and you will very quickly see that yes, you need transparency, wherever you get a lot of overlaps. The only other option would be for Therion to implement its own opacity engine and output a computed rendering, and I doubt anyone here wants to implement that.
We can gzip the files after creation but a foo.PDF.gz is a lot less handy than a foo.PDF with internal compression.
And Windows users would not know what to do with it. Does it take much space to bundle the fonts? Those can be extremely large in some cases (though I guess that part is simplified by Therion using just one font). If those are bundled, isn't it pretty much self contained? I personally see no need to remove compression. It's not like a PDF viewer is going to somehow lose support for it in future. Browsers have not dropped support for deflate or gzip. If anything, viewers (for as long as PDF viewers exist) will gain support for things like that over time, rather than losing it.
http://texdoc.net/texmf-dist/doc/latex/pdfx/pdfx.pdf Martin
22. 10. 2020 v 18:30, Tarquin Wilton-Jones via Therion <therion@speleo.sk>:
we need tranparency (I think)
Render your survey to SVG, and you will very quickly see that yes, you need transparency, wherever you get a lot of overlaps. The only other option would be for Therion to implement its own opacity engine and output a computed rendering, and I doubt anyone here wants to implement that.
We can gzip the files after creation but a foo.PDF.gz is a lot less handy than a foo.PDF with internal compression.
And Windows users would not know what to do with it.
Does it take much space to bundle the fonts? Those can be extremely large in some cases (though I guess that part is simplified by Therion using just one font).
If those are bundled, isn't it pretty much self contained?
I personally see no need to remove compression. It's not like a PDF viewer is going to somehow lose support for it in future. Browsers have not dropped support for deflate or gzip. If anything, viewers (for as long as PDF viewers exist) will gain support for things like that over time, rather than losing it. _______________________________________________ Therion mailing list Therion@speleo.sk https://mailman.speleo.sk/listinfo/therion
On Thu, Oct 22, 2020 at 6:39 PM Martin Sluka via Therion <therion@speleo.sk> wrote:
Hi, this unfortunately requires LaTeX, while Therion uses Plain TeX. I'm afraid I don't see any viable option to produce PDF/A directly from Therion without reimplementing the functionality of that package in Plain TeX. Martin
participants (7)
-
Benedikt Hallinger -
Bill Gee -
Martin Budaj -
Martin Sluka -
Tarquin Wilton-Jones -
Wookey -
Xavier Robert