in-place editing the right way (hint: don't use sed or perl)

So you have a text file you want to edit via a bash command-line script and you do some quick Google searching and find practically everyone agreeing that you should use sed or perl.  Great!  So you plop their example code into your script and cross your fingers.  Well, 99% of the time it works and you move on with your life a little happier.  Unfortunately, 1% of the time you get something that looks like:

sed: couldn't open temporary file /your/text/file/location/sedxV3rms: Permission denied

Hmmm, that's odd.  After digging a little more you discover you have write permission to edit the file but not write permission on the file's directory and sed and perl both create a temporary file to make the edits and then try to replace the original file with the temporary file upon save (which your folder permissions won't allow).  The easy fix is to update the folder permissions to give the user (or user's group) write permissions.

What if (for security, policy, etc.) reasons, you can't (or don't want to) change the folder permissions?  Is there another way?  It turns out there is.  Unfortunately, it requires using a rather arcane tool called ed.

Ed is a command-line text editor that, unlike nano or vi, accepts edit commands, rather than the file contents, as input.  A good overview can be found here.  Great, but how do I fix my problem?  Basically, you pass a number of text commands in sequential order to modify the file truly in-place (via memory buffer).

For example, to replace all the instances of "foo" with "bar" you would run:

printf "%s\n" ',s/foo/bar/g' wq | ed -s myfile.txt

Yeah, I know - ugh!  Don't worry, it's a little easier to understand if you break down the command piece by piece:

1. ed will open the file (myfile.txt) and suppress output (-s) so it doesn't interfere with the rest of your script logic
2. printf will loop through each argument (highlighted in green and purple) and pass (pipe) them to ed as individual line commands (i.e. commands separated by a newline return)
3. the first command it sends ed (highlighted in green) is a regular expression that says to look through the entire document (indicated by the comma right before the s — you can also use the longhand equivalent 1,$)  and find every string instance of "foo" and replace it with the string "bar" (if you only wanted to change the first occurrence, you would omit the g)
4. the final command it sends ed (highlighted in purple) tells ed to save the changes (w) and quit the program (q)

....and that's just the tip of the iceberg!  Here's a more complicated example:

printf "%s\n" '/foo/--a' 'bar' . wq | ed -s myfile.txt

1. the first command it sends ed (highlighted in green) tells it select the first line it finds that has the word "foo" (/foo/), go up two lines (--), and switch into input mode (similar to vi switching into insert mode)
2. the next command (highlighted in red) tells ed to enter the text "bar" as a new line
3. the next command (a single period highlighted in blue) tells ed to exit input mode
4. the final command it sends ed (highlighted in purple) tells ed to save the changes (w) and quit the program (q)

With a little practice (and an ed cheat sheet handy), you'll be scripting file edits in no time (with the added bonus of being lightning fast and avoiding permission issues).


Reference:

• http://backreference.org/2011/01/29/in-place-editing-of-files/