Coppieters, New Zealand

While I was working on a demo for TextStitch and InDesign Data Merge, I bumped into a weird phenomenon: some of my sample Data Source files were not recognized by InDesign.

I had created a small table with some data in Excel on my Mac, and I exported that file into CSV UTF-8 (Comma Delimited) format, using the File – Save As… menu.

When I tried to use this file as a Data Source in InDesign Data Merge it complained the file was not suitable.

Strange!

I then went back to Excel and exported the same spreadsheet as Tab Delimited Text. And that worked just fine!

I also tried saving the spreadsheet as Macintosh Comma Separated Values and that worked fine too.

I then compared the two CSV files using my text editor (I used BBEdit), and at first glance, they looked identical.

Why would InDesign Data Merge accept one, and not the other?

Notice the difference? The secret is the BOM character. BOM stands for Byte Order Mark.

These are three invisible characters at the start of the file: 0xEF, 0xBB, 0xBF, to allow other programs to more reliably guess a file as being encoded in UTF-8.

You might notice the two files also have different line endings (CR vs CRLF) but that is not a problem for Data Merge.

Workarounds

If you have a file that gets refused by InDesign’s Data Merge, you can work around the issue in many ways.

You could re-export the file differently if the source data is available to you. If there are accented characters in the data, you might want to use UTF-16 if it is available as an export option.

Or you can use a sufficiently capable text editor (like BBEdit or NotePad++) to open and re-save the file without the BOM.

On Mac, you want to re-save the file using the MacRoman encoding, or UTF-16 instead of UTF-8.

On Windows, you want to re-save the file using Windows-1252 encoding or UTF-16 instead of UTF-8.

This is because InDesign’s Data Merge does not handle UTF-8 encoded text.

For diagnosing these issues, I often use a hex editor program; using a hex editor, the BOM character is easy to detect and delete.

Below some screenshots from Hex Fiend

It’s not going to take my job

I’ve been using ChatGPT (and Copilot and similar) a fair bit while doing software development, and generally speaking, it makes me more efficient.

However, the more I use it, the less I am worried that some AI will ‘take my job’.

The more I use it, the more I see it as a fancy search engine.

It’s great for learning new stuff: if I need to get up to speed in a software development environment that I know very little about, it’s a great help.

However, as soon as the going gets tough, it becomes a confident bullshit generator.

Every single time I ask it a slightly more complex question, the answer seems to make sense, yet the answer is just wrong. Or it gives me a naive, impractical or overly complex solution.

I always have to carefully scrutinize what it came up with.

Me: ‘Hey, wait a minute: what happens if this or that situation arises? Will your solution actually work’?

And then ChatCPT will say ‘You’re absolutely right, well spotted, good job! Yeah, that would not work, eh. Here’s a different incorrect solution’.

It sure knows how to flatter.

With some coaxing ChatGPT will provide me enough raw material to work it out, but it cannot really fit all the pieces together. Keeping things as simple as possible but not simpler – that’s way beyond what it can do.

It’s a search engine, and it rehashes stuff it found on the internet.

Any time some real thinking is needed, it becomes a confident bullshit generator.

Engage Second Gear

While I am working, I feel that my brain has two gears.

First gear is ‘stuff I know off the top of my head’.

Second gear is ‘stuff I need to do some thinking about before I answer’.

In my normal day I’ll often need to switch into second gear, and do some thinking. Switching into second gear takes mental effort.

Depending on the complexity of the problem at hand, I might spend whole days stuck in second gear. I might need to go on long walks, mumbling all the way, brain in second gear, before I eventually figure something out.

The danger with ChatGPT is that it often allows me to comfortably remain in first gear, and not engage second gear as often.

For me, the challenge ahead is to make sure second gear gets used often enough, or else risk losing the ability.

So, I’d say: stay sharp, stay inquisitive, stay sceptical and make sure to have a big jar with salt at the ready. You’ll need it!

InDesign’s native file format (.indd) has a number of drawbacks, some of which can be worked around by way of an alternative file format, IDML (.idml).

IDML stands for InDesign Markup Language. It is an XML-based file format for InDesign.

You can think of an IDML file as a sequence of step-by-step instructions for InDesign. A little bit like

"Create a new document, facing pages, A4 page size"
"Start page numbering at page 2"
"Create a text frame on page 2"
...

but with much more detail and accuracy.

Opening an .idml file will make InDesign follow instructions and rebuild a brand new document from scratch.

InDesign .indd Versioning Issues

For the sake of argument, I’ll be referring to InDesign Old as the ‘older’ version and InDesign New as the ‘newer’ version. E.g. Old could stand for ‘2024’ and New could stand for ‘2025’, or something similar.

Some of the drawbacks of the native .indd format are that it is version-specific, not backward-compatible, and only partially forward-compatible.

.indd is a binary format, and between different versions of InDesign, the bits and bytes in the file are organized in slightly different ways, to allow for storing data related to various new features that have been introduced into InDesign New.

Not backward compatible

If you create a document using InDesign New, and save it, you won’t be able to open the resulting .indd file with InDesign Old.

InDesign Old is old, and hence ‘unaware’ of any new features that InDesign New has and cannot cope with the additional bits and bytes that have been inserted to cater for InDesign New.

Partially forward compatible

InDesign New can open files created by InDesign Old, but the compatibility is not quite 100%.

Most people don’t notice the occasional slight oddity in how InDesign handles File – Save and File – Save As… when it comes to opening and saving documents, but being aware of what’s going on will help you make sense of some odd behavior in InDesign.

When you re-open a document created by InDesign Old in InDesign Old, i.e. the same version, the File – Save menu item will be available. Once the document is open, it remains associated with the .indd file on disk, and File – Save overwrites the .indd file with the updated document.

When you open a document created by InDesign Old in InDesign New, it’s different: the File – Save menu item will be not be available.

Once the document is open in InDesign New, it is not associated with the .indd file on disk. You are forced to use File – Save As… to save the document.

You’re free to use File – Save As… to overwrite the existing .indd file, and doing so will upgrade the existing .indd file from version Old to version New. After that, you cannot open that .indd file with InDesign Old any more.

All of this because rather than opening the document, InDesign New has rebuilt a brand new, lookalike document from scratch. This brand new document is not tied to the .indd file that was created by InDesign Old. That’s why you need to use File – Save As… and cannot use File – Save.

IDML Solves Versioning Issues

Because an .idml file is just a sequence of instructions, we can go back to an older version of InDesign. The .idml can be created from InDesign New and interpreted by InDesign Old.

When InDesign Old comes across any instructions related to new features from InDesign New, InDesign Old will simply ignore any instructions that it does not understand.

And if the document created in InDesign New did not use any of the features that are specific to only InDesign New, opening the .idml file in InDesign Old would not cause anything to be lost in translation.

Note that opening an .idml file will also force you to use File – Save As… rather than File – Save: after InDesign has worked its way through the instructions in the .idml file, you’re left with a brand new document and the File – Save menu item is not available.

IDML does not retain text flow

One of the drawbacks of the way IDML works is that things like the exact text positioning, hyphenation and justification and other features are not controlled by the .idml file.

Given the same set of threaded text frames and the same text, there might be a discrepancy between how the text flows in InDesign Old vs. InDesign New and the same .idml document can end up looking different when opened in different versions of InDesign.

The exact same instructions in the .idml file can lead to different end-results in different versions of InDesign.

How it works

Diving a bit deeper, you can see IDML is a compressed .zip archive file which contains a collection of XML files, each XML file describing different aspects of the document. There are XML files with stories, styles, spreads…

These files, taken together, contain the instructions needed to recreate the InDesign document.

That also means that editing any one of these XML files would change the corresponding aspects in the InDesign layout. This allows software developers to inject new or modified data into an InDesign template that is saved into the .idml file format.

Round-Trip Workflow

One practical way to leverage IDML is through a round-trip workflow: convert an IDML file to plain XML, edit the XML, and bring it back into InDesign.

Unzipping an .idml file to inspect it is easy: simply change the file name extension of the file from .idml to .zip and you will find that the file is now just a .zip archive with a bunch of XML files and folders in it.

The reverse operation is not as straightforward: converting a folder with a collection of files and folders back into an .idml file is more finicky, as there are some quite specific requirements for the ‘zipping’ process.

The Oxygen XML Editor is able to do ‘direct edits’ into .idml files: you can browse ‘into’ the .idml file and experiment with tweaks to the XML files.

https://www.oxygenxml.com

Another option is the free eCanCrusher tool: you can drag/drop an .idml file onto the app to ‘unzip’ to an .idmlf folder, and you can then tweak the XML files.

Then revert back to an .idml by drag/dropping the unzipped folder, and eCanCrusher will convert the whole folder back to an .idml file.

eCanCrusher was originally only meant for handling .epub files, but as it so turns out, .epub and .idml files are very similar in how they are actually just .zip archives with a bunch of text files in them.

https://www.docdataflow.com/ecancrusher/

Next Steps

If you’re interested in finding out more about .idml there is a lot of material within reach – search engines and the Adobe Forums are your friend.

If you’re interested in automating part of an InDesign- or InDesign Server-based workflow by way of IDML, please reach out to [email protected] if you’re interested in a hands-on custom training, or want some help (‘training wheels’) as we help kick-start your project.

If you find this post to be helpful, make sure to give me a positive reaction on LinkedIn! I don’t use any other social media platforms. My LinkedIn account is here:

https://www.linkedin.com/in/kristiaan

Divide and conquer isn’t just for politics; it’s a solid approach for diagnosing or reporting bugs.

TD;RL

When reporting a bug or asking for tech support, make sure to provide ample information. Providing this info from the outset can greatly reduce the amount of ’email ping-pong’ or ‘forum ping-pong’ and lead to a quicker resolution.

The Situation

The information described below is not specific to InDesign, but as I live and breathe a lot of InDesign, I’ll use InDesign as an example application.

Imagine you are working with Adobe InDesign, and you perform a certain operation, and InDesign crashes (or hangs or exhibits some other unexpected behavior).

You restart InDesign, recover the document, try the same operation—and it happens again.

You might not know it, but you’re in luck! You found a repeatable problem.

As we all experience all too often, many crashes and hangs are not repeatable. Something odd happens, and when you want to show it to someone else, it works just fine.

Those kinds of non-repeatable bugs are the most frustrating: if you cannot reliably make the problem occur again, reporting it to the software developer (for InDesign that’s Adobe) will probably do little good. It’ll be classified as ‘cannot repeat’ and that’ll be the end of it.

A crash – that’s bad 🙁 .
A crash that is repeatable – that’s good 🙂 !

Mind you, non-repeatable crashes are not the end of the story. They might become repeatable when they happen frequently enough. It’s a matter of remaining vigilant and consciously take note of as many details as possible each time it happens.

After it happens a few times, if you pay attention, you might start noticing some kind of pattern that you did not notice before.

For example, ‘it’s always when I hold the Opt key while doing this or that’, or ‘it’s always when I place an image from that server’.

Keep your eyes and ears peeled, and with a little luck you might figure out what triggers the problem.

And eventually, the crash might become repeatable, and you can write out a little step-by-step process to make it happen.

“Open this document, while holding your breath. Then press the Opt key three times while shouting out ‘abracadabra’.”

Whittling down

Crash! Try again. Crash!

You might now be inclined to rush to the developer’s bug report site (e.g. Adobe’s User Voice), and file a bug report, or send an email.

However, that’s probably not going to do much good.

Yeah, it’s better than not filing a bug report at all, but you can do better, and improve the odds of getting a solution to the issue.

First of all, you might browse around and see if there are any other bug reports from other users that match the issue at hand.

If not, whittling down the crash will allow us to get a better idea of what the root cause might be, and in the process, you will often find a workaround.

E.g. “it crashes if I place a TIFF that’s more than 1024×1024 pixels” and then you find out that the issue does not occur if you use PNG instead of TIFF… Great! You’ve found a workaround.

Whittling down is the process of eliminating irrelevant factors from the context and collecting data points that will help finding a workaround and filing a good bug report.

I always put a lot of effort trying to whittle down any bug reports I file.

And as a software developer myself, I tremendously appreciate when one of my customers makes the effort to whittle down the problem before emailing me.

The worst nightmare for a software developer is to get an email from a frustrated and stressed user, often in all caps: “your software does not work”. It’s frustrating not being able to help them because of a total lack of context. What software? What platform? What version? What does not work?

Being forthcoming with info always helps getting the problem resolved faster.

First Steps

The first steps in whittling down involve platform, version and preferences.

Platform

What platform are you on – Mac? PC? x86 or ARM processor? Windows 10? Windows 11? How much memory… These parameters might or might not be important, and if you can try to figure out whether the problem is tied to the platform used or not, that’s very helpful.

Is it possible to try things out on a Mac instead of a PC? Or use InDesign 2025 instead of InDesign 2024? If you have a colleague or friend with a different platform, have them try it out and see if they can get the same repeatable crash. That kind of info is invaluable to help diagnose bugs.

Version

The second parameter is the version of the software you’re using. Nowadays with Creative Cloud it is fairly straightforward to figure out whether the problem is version-dependent, because the Creative Cloud Desktop application gives you access to older versions. You can do a bit of sleuthing yourself.

While you’re at it, make sure you have tried the latest version.

And if the problem turns out to be version-related, figuring out where between the successive version numbers the issue comes or goes is also invaluable.

E.g. “It crashes in InDesign 18.5 but not in 18.4” – that’s a solid gold data point!

Preferences

Many applications, including InDesign, store ‘semi-permanent’ preferences and cache files on your computer, and many bugs and crashes are tied to ‘something wrong in the preferences’.

You can try resetting your preferences (e.g. with InDesign, hold Cmd-Ctrl-Opt-Shift or Shift-Ctrl-Alt while re-launching the app) but that is quite dreadful. You’ll lose all your finely honed settings and will have to re-configure the application from scratch. Worse, there is also persistent stuff that is not cleared by a preference reset.

Instead, you can do a quick test. You do need to be master of your computer (i.e. have administrative privileges). If you are working on a computer that is managed by an I.T. department, then you’ll need to get the I.T. department involved to run this test for you.

Create a brand new user account on your computer. For example, my day-to-day account on my Mac is kris. I will then proceed to create another account, kris2. kris2 gets the same privileges as kris, but when I log out of kris and into kris2 I get a ‘clean slate’ to experiment with.

While logged in as kris2 I then try the repeatable crash and if it does not happen any more, I know it’s an issue tied to something in the ‘semi-permanent’ preferences for the kris account.

Knowing the results of this test is another important data point, and helps me decide whether there is any point in going through the pain and frustration of resetting the preferences in the kris account.

After it has served it’s purpose, I can delete the kris2 account.

Plugins

If you have any enhancements installed, like plug-ins or plugins or custom panels or scripts, try to see if the issue is related to the enhancement.

If possible, uninstall the enhancement and try again. Does the problem disappear?

Note: this does not mean that the enhancement causes the problem. You have to be very careful not to confuse correlation with causation.

In other words, the enhancement might be an innocent bystander, involved in causing the crash, without actually causing the crash.

Just recently, a user reported an issue with our free TextStitch plug-in.

I was in luck and the user was willing to do some whittling down on their end, and quickly came to the conclusion the core issue had nothing to do with TextStitch. They could also make the crash occur without any plugins installed.

Turns out, InDesign has a bug where creating a text thread between two text frames, in certain cases will cause an immediate crash. This crash also occurs when manually linking the frames, while TextStitch is not even installed.

TextStitch was but an innocent bystander. Because TextStitch needed to automatically create threads, it inadvertently tried to create the ‘impossible’ thread between two text frames, causing InDesign to crash.

The obvious, yet incorrect diagnosis was to blame TextStitch, where the real bug was in InDesign.

Divide and Conquer

The next part of the process is to try and write up a simple step-by-step process that any tech support person or developer can follow and see the crash for themselves.

This often involves a particular document or image.

In the case of InDesign the document that is involved in the crash might be large or confidential, and you cannot attach that particular document to your bug report.

Try a New Document

The first thing to try is to build a brand new document, and see if you can make the crash occur with it.

Create a new document, put in some text, maybe some images. Can you make the issue occur? If so, you’ve got a good sample document to attach to the bug report.

Try Copy-Pasting stuff

If a new document does not do the trick, try copying a few elements from the problematic document into a brand new document.

I’ve often found that ‘trouble travels’. Sometimes the issue is ‘attached’ to a text frame or image frame or style sheet, and copy-pasting stuff into a new document is enough to set up a small new document that also exhibits the issue.

Try the IDML round-trip

In the case of InDesign, we also have a secret weapon: the IDML file format.

Not many people know this, but behind the scenes, an IDML file is really a script.

What InDesign does when you save a document as IDML is to convert your document into a series of instructions: “Create a new page. Place a text box of size xyz at position pqr. Insert the text ‘bladibla’. Format the ‘bla’ with Comic Sans 12…”

An IDML file is just a cookbook with instructions on how to re-create your document from scratch.

When you open an IDML file, InDesign will execute all the instructions, and rebuild a brand new document from scratch.

That is why older versions of InDesign can open IDML files created by newer versions of InDesign: they will simply follow the instructions.

Normal .indd files are like elefants, and they have long memories, and hold grudges.

Often .indd files have passed through many versions of InDesign (e.g. originally created in CS6, then upgraded to CC 2019, then upgraded to 2024).

They hold on to a ‘memory’ of that chequered history. If one of those past InDesign versions had bugs, those bugs might leave weird problems embedded in the .indd file.

IDML to the rescue: saving a document into the .idml format, then re-opening and saving gets rid of all that past history: we’re rebuilding the document from scratch, and all those repressed memories disappear.

Round-tripping your document through IDML is always a good move!

Finally, Divide and Conquer

If we’ve gotten to this point, nothing has helped, and we’re still stuck with a document that has some issue somewhere.

The next step is to do ‘successive halving’. Do this on copies of the document (preferably after an IDML round trip). Keep the original safe!

What comes is not scientific: just delete a bunch of stuff. You might need to try different deletion strategies until you hopefully hit the right one.

Open the starting document, and delete about half of the content.

This might be ‘half of the spreads’, or ‘half of a long story’, or ‘half of the styles’.

The main thing is that you delete a lot of info, and make the document about half of the original’s size. If you delete pages, you might also need to delete text from long stories – otherwise, there might be a lot of overset text in the ‘halved’ document.

Then save the document. That’s ‘exhibit-A’

Open the same starting document again, and delete the other half of the content.

Save this document. That’s ‘exhibit-B’.

Now try the repeatable crash on both exhibit-A and exhibit-B.

If the problem disappeared in both of them, go back and try a different size reduction strategy (e.g. delete text instead of deleting pages).

If the problem has moved into either exhibit-A or exhibit-B or still exists both, pick one of the halved documents that has the issue, and repeat the halving strategy – this will give you exhibit-A-A or exhibit-B-A or whatever.

Keep on halving, retracing your steps, halving… as needed and try to create the smallest possible document that still causes the crash.

Once you get to that point, it often becomes apparent what the underlying cause might be. You might find it’s related to text reflow in particular text frames, or a particular image frame with particular settings.

By now, with a little luck, you might have figured out how to work around the bug and you now have a small document and a concise step-by-step procedure to make the crash happen.

You’re ready to file a bug report that has better odds of being actioned!

Quick Checklist

• Exact steps you took before the crash
• Platform (Mac/Win/Operating System…)
• Version (InDesign 20.0, InDesign 18.5,…)
• Did you try turning it off and on again
• Did you try resetting your preferences
• Attach screenshots
• Report any error messages
• Provide as much info as you can. Context is everything!
• Attach a reduced-size sample document

For Creative Cloud apps, head to:

https://helpx.adobe.com/x-productkb/global/how-to-user-voice.html
https://community.adobe.com/

If you find this helpful, make sure to give me a positive reaction on LinkedIn! I don’t use any other social media platforms. My LinkedIn account is here:

https://www.linkedin.com/in/kristiaan/

I had a bit of a play with Google’s NotebookLM – trying to generate a self-referential conversation.

Here it is; have a listen, have a think!

And another one:

Soxy, bless its soul, was an application we built to fix a frustrating issue that occurs when a workstation has multiple versions of an application installed.

As it so turns out, Soxy might return under the guise of PluginInstaller (https://PluginInstaller.com)…

The issue: you might be using the Adobe Creative Cloud and have multiple versions of an app installed.

For instance, having both InDesign 2024 and InDesign 2025 installed, or InDesign 2025 and InDesign 2025 Debug will cause unexpected double-click behavior in the Mac Finder or Windows Explorer.

What happens when you double-click an .indd file? Often, the wrong app will launch and open the file (or refuse to do it). Gah!

Soxy would intervene in the double-click process in Finder or Explorer. It would automatically detect the version of a double-clicked .indd file and route it to the correct app.

E.g. if you double-click an .indd file created with InDesign 2024, it would open that file in InDesign 2024, even if InDesign 2025 was also installed and running.

Over time, the versioning issues became less of a problem for most users after Adobe Creative Cloud became subscription-based. The majority of users now keep only the latest version of an app installed. There was little demand for Soxy, and we had to put it out to pasture.

These days, I am working on PluginInstaller most of the time, working on a unified installing experience.

As it so turns out, this might lead to a resurrection of Soxy.

Where Are The Apps?

The next big task ahead of me is to enhance PluginInstaller to also handle C++ plug-ins smoothly.

It’s a major work in progress. The current version will install and uninstall InDesign ExtendScript and CEP panels with a click of a button.

Contrast to that, C++ plug-ins still need to be manually dragged into a Plug-Ins folder. The current version of PluginInstaller will handle activation and licensing, but installing is not yet supported.

In my view, installing any enhancement to any app in the Adobe Creative Cloud should work the same from a user perspective. The user does not need to know what technology drives the enhancement (script, C++ plug-in, CEP extension, UXP extension…). So that’s the goal I am working towards.

Part of my current effort is to figure out where to install ‘stuff’, and that turns out to be a little more complicated than it might seem.

For example, if you have an Applications folder that looks like this:

If I simply say a ‘C++ plug-in for InDesign’…, then there are 11 places in this screenshot where that plug-in might need to go.

Factors that play into the calculations PluginInstaller will need to make:

• Some apps have version requirements: e.g. InDesign C++ plugins compiled for InDesign 2024 don’t work with InDesign 2025. It’s been a while since I worked on Photoshop and Illustrator plug-ins, but I seem to remember that plug-ins for Illustrator or Photoshop have some ‘version flexibility’, so the version requirements are not always clear-cut.
  Also, with InDesign, we all still remember the 16.x issues, where plug-ins became sub-version-dependent (e.g. 16.0 being incompatible with 16.3).
  I need to make sure PluginInstaller is resilient enough to handle this situation when this happens again.
• Some plug-ins are ‘cross-app’ – C++ plugins for InDesign might (or might not) also work with InCopy or InDesign Server or vice versa.
• Some apps have ‘year’ versioning – 2023, 2024, 2025. Others do not (e.g. Lightroom Classic, UXP Developer Tools,…).
  I also prefer to accept alternate version numbers – e.g. InDesign 2025 can also be referenced to as InDesign 20.0 or InDesign CS 18 or InDesign CC 2025.
• InDesign has Middle Eastern versions, which have some differences with the regular versions.
• Some apps have debug versions which are mutually incompatible with regular versions.
• Install locations for ‘stuff’ use different kinds of logic depending on the technology used.
  * CEP panels can be shared between multiple app versions (e.g. InDesign 2024 and InDesign 2025 can run the exact same CEP panel code from the exact same install location, CEP manifest allowing).
  * InDesign scripts are installed in a version-dependent location but are shared between multiple app executables of the same version (e.g. InDesign 2025 Debug and InDesign 2025 share the same Scripts folders, but InDesign 2025 Middle Eastern does not).
  * C++ plug-ins are handled on a ‘per app executable’ basis.

Briefly put: ‘it’s complicated’.

Building A Local Knowledge Base

In order to get this functionality into PluginInstaller, I am working on a queryable local knowledge base.

This is a data structure that keeps track of which apps are installed where, and where their install locations are (e.g. Plug-Ins folder, ~/Library/Application Support,…).

PluginInstaller can scan the local infrastructure and has an internal API that can be queried by the installer module to figure out what options there are to install a certain ‘thing’.

E.g. for an InDesign/InCopy/InDesign Server plug-in, we need to pass into the API:
– for what version
– debug or non-debug
– InDesign, InCopy, InDesign Server, or a combination thereof

Whiff Of Soxy

I’ve only just started working out how this is all going to work, but as I was working on this it felt strangely familiar.

And then I suddenly realized: I’ve done very similar stuff before, when we were building Soxy.

And once this data structure in PluginInstaller is properly working, re-implementing Soxy within PluginInstaller will be quite straightforward.

Creating a smooth installation experience is first on the to-do list, but a resurrection of Soxy might follow some time after that!

Recording this for my own future reference…

Occasionally, after installing an update to the InDesign SDK, my Mac will refuse to run odfrc-cmd which is a command needed in the build process. The fix is simple. Start a Terminal window and execute the following commands (<SDK> is the path of the decompressed SDK):

cd <SDK>/devtools/bin
xattr -dr com.apple.quarantine odfrc-cmd  

If you find this helpful, make sure to give me a positive reaction on LinkedIn! I don’t use any other social media platforms. My LinkedIn account is here:

https://www.linkedin.com/in/kristiaan

I wanted to share a tiny little trick I recently tried…

It’s nothing much, more like a ‘doh…’, but I had never tried this approach, and had not fully expected it to work!

I tried it with InDesign and InDesign Server, and the odds are good that the same trick will work with any Adobe app that has an ExtendScript scripting engine.

Non-invasive tweaking

The issue at hand was that I needed to ‘wrap’ an existing standalone ExtendScript, and embed it into a controlled environment.

Part of the functionality of this environment is to keep tabs on the logging output of the embedded scripts.

The issue was that the standalone ExtendScript was in part using calls to $.writeln() for some of its logging.

There is more than one way to skin a cat. I could have scoured the source code, do a search-and-replace on $.writeln(), and set up a replacement bottleneck function (e.g. something similar-looking, like function $_writeln() or something) and replace all calls into $.writeln() with $_writeln(). But that approach felt quite invasive.

Instead I decided to try and re-direct $.writeln() by injecting a replacement function.

This allows the standalone script to remain unmodified and continue calling $.writeln().

Injecting a replacement for $.writeln()

If the script is run as a standalone, $.writeln() works normally.

When the same script instead runs within the controlled environment, some ‘outside’ wrapper code will initialize things, inject a replacenent, then invoke the script.

The script then blithely continues to call $.writeln(), which is now redirected to some API provided by the controlled environment.

During initialization, all you need to do is to run something akin to the following function:

function divertWriteln() {
    if (! $.systemWriteln) {
        function customWriteln(msg) {
            FRAMEWORK_API.logTrace(msg);
        }
        $.systemWriteln = $.writeln;
        $.writeln = customWriteln;
    }
}

This function installs a different handler for calls to $.writeln(), and the calling code is none the wiser.

InDesign Server-Specific

In the InDesign Server environment, this trick can also be used to dynamically re-direct $.writeln() which normally outputs to the debugger output window. By redirecting we can convert that into a call to alert() which instead outputs into the InDesign Server Terminal window.

Something similar to:

function divertWriteln() {
    if ("serverSettings" in app && ! $.systemWriteln) {
        function customWriteln(msg) {
            alert(msg);
        }
        $.systemWriteln = $.writeln;
        $.writeln = customWriteln;
    }
}

If you find this helpful, make sure to give me a positive reaction on LinkedIn! I don’t use any other social media platforms. My LinkedIn account is here:

https://www.linkedin.com/in/kristiaan

I think I found another obscure bug in ExtendScript.

This bug affects any ExtendScript code that is passed through eval().

eval() spits the dummy when you try to evaluate any JS code that contains literal strings that contain the literal Unicode characters U+2028 (line separator) or U+2029 (paragraph separator).

This can be worked around by encoding these characters by their escaped equivalents, \u2028 or \u2029.

I verified the 16-bit Unicode range, and these are the only two characters that cause problems. See

https://github.com/zwettemaan/ESON/blob/main/findBadCodes.jsx

It all started because I was curious and wondered what it would take to use eval/uneval as the basis for implementing JSON.parse/JSON.stringify.

json2.js

One of the most common solutions is to use Douglas Crockford’s library, JSON-js (aka json2.js).

https://github.com/douglascrockford/JSON-js

You can readily //@include this file into your ExtendScript code, and gain access to JSON.parse() and JSON.stringify().

This module is time-proven, and performs a proper parse of the input data, which is useful if you need to ingest JSON data from untrusted sources.

This protects your script from injection attacks, where a malicious actor crafts a ‘fake’ JSON file which contains some executable JavaScript code.

Executable JS code is not proper JSON, so json2.js will simply refuse to parse and will throw an exception. That’s a great feature!

A disadvantage of using json2.js is that it is not very fast when used with ExtendScript.

Especially when parsing larger and larger amounts of JSON data, the time needed will start to balloon and it will slow to a snail’s pace, or even slower than that: the relation between data size and slowdown is not linear. I’ve not properly benchmarked it, but I suspect it might be an exponential relation.

The best way to get around the slowness is to use a C++-based alternative, rather than a ‘pure JS’ solutions like json2.js.

When it comes to ingest large, multi-megabyte JSON files, C++ code will eat those in a tiny fraction of the time it takes json2.js to slog through them.

Note 2024-09-17: Marc Autret (Indiscripts) sent me some feedback on my blog post and pointed out a bunch of problems with json2.js.

https://www.indiscripts.com/

If you need a reliable JSON module for ExtendScript, make sure to look at Marc’s idExtenso

https://github.com/indiscripts/IdExtenso

eval()

Another alternative is to use the built-in eval() as a ‘quick and dirty’ replacement for JSON.parse().

JSON is a subset of JavaScript, and simply handing over some JSON data to eval() works fine because eval() will treat it as executable code, and evaluate the JSON.

Advantage: this is much faster than json2.js.

Advantage: eval() can process JSON-C (i.e. JSON with comments).

BIG disadvantage: eval() has some serious problems, and it should only be used in environments where the JSON data comes from a trusted source.

eval() is problematic, because it does not verify that the JSON data is just that, data.

That means that there is a risk that some malicious actor would to be able to ‘inject’ some fake JSON file with executable JS code into your workflow.

In short: in a pinch, eval() can be used as a quick-and-dirty replacement for JSON.parse(), but I think it’s better to be safe than sorry, and avoid this in production code.

uneval()

ExtendScript is ‘old time JavaScript’, and as such it still supports the uneval() function.

uneval() will produce output that is similar to JSON, but not quite.

var o = {
  a: 1,
  b: [1,2,3],
  c: {
    d: 9,
    e: "ab\u0101c"  
  }
}
uneval(o);

will produce:

 ({a:1, b:[1, 2, 3], c:{d:9, e:"abāc"}})

Proper JSON would be:

{"a":1,"b":[1,2,3],"c":{"d":9,"e":"abāc"}}

Fake it till you make it

For the heck of it, I managed to create a usable implementation of JSON.parse/JSON.stringify based on eval/uneval, and it seems to work fine.

https://github.com/zwettemaan/ESON

ESON.stringify() will transmogrify the output of uneval() into proper JSON. It is slowed down a bit because it needs to make sure that U+2028 and U+2029 are properly encoded.

If you are 100% certain that U+2028 and U+2029 never occur in the data you’re processing, then ESON.stringify() can be sped up a fair bit by omitting the check for the ‘bad codes’.

ESON.parse() is a lot faster than JSON.parse(), but should be avoided because of its insecure nature.

There is also a bunch of benchmarking code: it will generate random large objects, and run them through stringify/parse in order to compare json2.js with ESON.

Use at your own risk!

If you find this helpful, make sure to give me a positive reaction on LinkedIn! I don’t use any other social media platforms. My LinkedIn account is here:

https://www.linkedin.com/in/kristiaan

Traditional Model

Most InDesign Server workflows that I’ve worked on are based on reusable ‘hot instances’.

In this model, there is a monitor program of sorts, which pre-emptively launches one or more instances.

These instances are then initially idling: they’re ‘hot’, ready to roll.

Jobs are coming in via a queue managed by the monitor program. When the queue is not empty and there is an idle instance available, the monitor program will assign the job to the instance by sending a script to the instance.

This can be achieved in a number of ways:
– OLE (Object Linking and Embedding)
– AppleScript/AppleEvents
– VBScript
– SOAP (Simple Object Access Protocol)
– InDesign Server startup script with a polling loop
…

Some info about these can be found in the InDesign Server documentation.

While an instance is processing the job, the monitor program has some means of tracking the instance and determine whether the job has completed.

There are multiple ways to do this; I like to use a ‘customized heartbeat’ approach, where the script code is instrumented to let the monitor know it is still running, and the monitor program will not mistakenly kill the instance if the script is merely taking a long time, but not hanging.

When the job completes, the instance goes back to idling until the next job is assigned by the monitor program.

The monitor normally also has some background communication channel with the scripts running inside the instance, so the monitor can detect crashes, deadlocks and freezes and act accordingly.

Well-written scripts should also handle some kind of ‘intermediate state’ management and a ‘roll forward’ where a long-running crashed job can be picked up at the point where it crashed, rather than needing to start over.

Run to completion

There is another approach to handling multiple instances.

In the ‘run to completion’ model, when the system is at rest, no instance will be running. All instances are ‘cold’, not yet started up.

When the monitor program receives an incoming job in the queue, it will cold-start a fresh new InDesign instance, and pass the job to the instance. This can be done via a startup script that picks up the head of a job queue.

The script runs to completion and the last task of the script is to quit the instance.

Having the instance quit is a simple way to inform the monitor program that the task has been completed, or that the job has crashed.

Advantages and disadvantages

Run to completion has a number of advantages.

Long running InDesign Server instances (‘hot’, reusable instances) have a habit of bloating: their memory footprint slowly grows and after a big job, some memory can remain ‘stuck’. As the bloat increases, garbage collection can introduce random ‘hiccups’.

With run to completion we need to worry much less about memory leaks and unreleased resources (e.g. files that remain open), because InDesign Server will cold start every job with a clean instance.

If the server machine has oodles of memory, the InDesign Server will never need to invoke garbage collection, and random ‘mumble mode’ due to garbage collection should be less of an issue: most jobs will be completed before InDesign Server would feel the need to run garbage collection.

Disadvantage of run to completion is that each job takes a few seconds longer because we need to cold-start IDS for every job, whereas hot instances are ready to go – no cold-start delay.

Hybrid model

A hybrid model combines these two models.

For fast response times (e.g. quickly rendering a layout that needs to be displayed in a web browser), hot instances are the obvious choice. These instances can then be restarted every so many hours to clear up accumulations of small memory leaks or resource leaks.

For long-running tasks, like paginating a 1000-page catalog with lots of reflowing content, cold instances might be better: after a long hard slog on a 1000-page catalog, it’s probably better to terminate the instance and use a clean slate for the next long-running task.

The monitor program can be made to handle both types of tasks.

I hope you found these ideas useful!

You can contact me at [email protected]. If you want to sharpen your InDesign and InDesign Server automation skills, talk to me about organizing a custom workshop.

If you’re stuck trying to resolve an intractable complex issue around InDesign automation, and you’re stumped, think of seeking my help. I have a proven track record of being able to resolve tricky issues and explain everything in a clear and concise way. I’ve been automating InDesign server for nearly 20 years, and I know a trick or two!

If you find this helpful, make sure to give me a positive reaction on LinkedIn! I don’t use any other social media platforms. My LinkedIn account is here:

https://www.linkedin.com/in/kristiaan