Learn to Shrug


Containerization Platform splintering


It was inevitable.  Docker's unprecedented growth was simply too big and too fast to manage.  It was only a matter of time before major contributors and admiring architects started envisioning a "better way" (aka "their way").  Currently, there seems to be a growing list of contenders since Docker released their first public offering in March 2013:

  • Imctfy (October 2013)
  • Terminal (unknown launch date -- at least as early as June 2014)
  • Vagga (released around July 2014)
  • Flockport (initial public offering August 2014)
  • LXD by Canonical (announced November 2014)
  • Rocket by CoreOS (initial public offering December 2014)

Note: I highlight Docker due to its current popularity, but container technology has been around for a long time.  For example, Parallels Virtuozzo Containers were introduced in 2001 [leveraged in Jelastic] and Solaris Containers were introduced in 2004 [leveraged in SmartOS].

AWS CloudFormation template for a VPC with NAT


I had trouble finding a nice generic CloudFormation template that would take the complexity and pain out of setting up a VPC with NAT so I made one myself.  Enjoy:

AWS VPC NAT issue with Debian


This is an incredibly obscure issue so I doubt it will help anyone, but in the rare chance that some poor soul is about to waste 3 days on this problem I hope to spare you the grief...

AWS VPC supports some advanced network topologies, including NAT.  Using a CloudFormation template similar to this one, I created a VPC with public and private subnets and a NAT instance to route internet traffic from the private subnet.  The setup worked great for Amazon Linux and Ubuntu AMIs but I needed a Debian image for my specific use case (yes, I know Ubuntu is based on Debian but it had issues with VirtualBox...).

Unfortunately, when I switched to the Debian image, my private subnet instances simply would not connect to the internet.  Like I mentioned, I wasted three days scouring the Internet and learning more than I ever wanted to about iptables.  Long story short, you need to run the following commands as root on your NAT instance (or add it to the NAT EC2 instance UserData section in your CloudFormation template):

echo 1 > /proc/sys/net/ipv4/ip_forward
echo net.ipv4.ip_forward = 1 > /etc/sysctl.d/ip_forward.conf
apt-get install -y iptables-persistent
iptables-save > /etc/iptables/rules.v4

Docker traps (and how to avoid them)


Trap #1

Dockerfile:  FROM ubuntu  (or fedora, centos, ...)   -OR-
docker run -i -t ubuntu  (or fedora, centos, ...)

Dockerfiles allow you to start from a base image (which is a good thing).  However, those base images are often very large:
  • ubuntu: 193 MB
  • centos: 236 MB
  • fedora: 374 MB
  • phusion/baseimage: 437 MB
Although Docker caches installed images so future containers based on the same FROM image will pull locally, that's still a big initial download for unreliable networks (e.g. WiFi, developing countries, etc), a lot of overhead for clusters managing load-balanced containers across servers/regions, increased storage space for regular docker backups, and (most importantly) they expose a much larger attack vector for hackers attempting to break into (or out of) your container.


If you need a named data volume container, use FROM scratch (which consumes a whopping 0 B of disk space).  Source: Michael Crosby, a member of the Docker Team  Update: Brian Goff shares an alternate opinion on this recommendation.

If you need to run a simple script, use FROM busybox:latest (2 MB disk space)

For everything else, use FROM gliderlabs/alpine:latest (5 MB disk space).  Alpine Linux is a security-oriented OS (often used for routers, firewalls, VPNs, etc.) that can run just about anything you need.  More info can be found on its Docker Hub page.  Update: Alpine Linux has some Java issues so I'm no longer recommending it as a good general use base image.  Update #2: now that Docker includes an official Alpine Linux image, an S6 wrapper is available, and Java 7 is supported, I'm now considering it again.  Update #3: Alpine Linux still has DNS issues that may be problematic for your use case.

Tiny Core Linux (7 MB, with lots of supported apps and the base for boot2docker) looks promising but no MySQL.  

Another promising option is SliTaz (9 MB) but their stable release is almost three years old and their English packages search page returns a 404.  

Ultimately, you'll probably need to compromise and follow Docker's official best practices recommendation to use FROM debian:latest (85 MB).  More info can be found on its Docker Hub page.  

Or, you can try FROM textlab/ubuntu-essential:latest (65 MB) as described here.  

If you want to standardize on Ubuntu and prefer a more "official" image you can use FROM ubuntu-debootstrap:latest (87 MB) which is an "official" Docker-sanctioned image but is not an official Canonical-approved Ubuntu image.  More info can be found on its Docker Hub page.

If you need an RPM-based solution, try FROM yamamon/centos7-zero:latest (137 MB) as described here.

P.S. Dray looks interesting (similar concept to Docker Nano, which is based on Buildroot).


Trap #2

Run everything as root.

This is obviously a bad security practice, but most Docker processes and images today run everything as root.  Not only is this dangerous for potential exploits launched from within the local container (like XSS, CSRF, shellshock, etc.) that manage to run arbitrary code, but can also allow hackers to break out of the container and achieve root on the host system (thereby allowing them to control all containers on the host and launch further network attacks).  For further reading:


The first step is to run your process as a regular unprivileged user.  See the official Docker best practices guide and the Dockerfile User documentation for details.  Note: the best practices guide also says "You should avoid installing or using sudo since it has unpredictable TTY and signal-forwarding behavior that can cause more more problems than it solves".  Also, for Alpine Linux, the command would be something like addgroup mygroup && adduser myuser -s /bin/bash -g mygroup

Furthermore, you should use a hardened Docker base image (like Alpine Linux).

Finally, you should use SELinux or AppArmor to separate containers from the host and each other.  Note: watch Pull Request #777 for progress of CoreOS SELinux or AppArmor support.  Update: CoreOS SELinux is supported in version 808.0.0!

FYI: As an aside, the recent Docker 1.2.0 release implemented --cap-add and --cap-drop capability so in theory you could do something like docker run -it --cap-drop=ALL --cap-add=SYS_TIME debian:latest /bin/bash to restrict all privileged permissions except the ability to set the system clock (no chown, useradd, etc.).  That said, Linux capabilities are murky and complex (for example, which capability allows adding users and groups?) and most of them allow privilege escalation anyways so I doubt this feature will be all that useful to regular admins.  Also, they added a new run flag called --privileged (disabled by default), which is a good step forward.

Conclusion: Even though Docker has passed the "1.0" milestone, it's not production-ready (unless you want to run a single instance in a VM without scale capabilities, container patch management, multi-tenant support, persistent data options, or full SDLC support*  ;)

...and then there's that whole PID 1 zombie problem...

* Update: Kubernetes appears promising to solve the aforementioned production limitations and LXD looks promising to solve the multi-tenant security concerns.

* Update #2: Snappy Ubuntu Core also looks promising for a multi-tenant security solution (but the nascent project still lacks a broad selection of "snappy apps").

* Update #3: Flynn has matured quite a bit since I last investigated it.  It supports a Postgres database cluster out of the box, which is a huge win over doing it manually with something like Crunchy Postgresql Manager (note: MySQL-to-Postgres converter can be found here).  Once they add persistent file storage (like GlusterFSLizardFS, or powerstrip-flocker) and get a better looking dashboard (like Cockpit or Rancher), it could be an awesome Docker PaaS leader.


Trap #3

Not including a tag when performing a pull or run.


Always provide a tag, even when the only option is "latest".  For example, instead of docker pull ubuntu you should run docker pull ubuntu:14.04

P.S. Use version number tags instead of distro code names.  "Trusty Tahr" may sound cute, but at a glance, ubuntu:14.04 is a lot more meaningful than ubuntu:trusty


Trap #4

Storing anything you care about exclusively in the container.

Valuable, persistent data (user content, databases, logs, etc.) should not be permanently and exclusively stored in a Docker container or VM host instance.  Containers (and the cloud VMs they're likely running on) are ephemeral by design, meaning everything gets wiped out when they restart or a new instance is provisioned. It's a paradigm shift from pets to cattle.  A lot of people are trying to find ways to keep data persisted inside and/or across ephemeral Docker containers, but here's the reality:
"if you put anything beyond your base OS on ephemeral storage, you are at great risk. That data could go away at any time. You can’t depend on it, so don’t use it unless you add in an additional form of redundancy at your own engineering expense. Data you care about belongs on block storage: it has built-in redundancy and improved availability, which ensure that the data you care about will be there when you need it."
     - Pete Johnson 


You have two options:

1)  Use a data volume or data volume container.  Essentially, you're mounting a directory from the host server to use within the Docker container.  When adding or modifying files in the container, you're transparently writing them to the host server instead.  Because of this, the data (within that specific mounted folder) persists when the Docker container stops.

Important: There's a subtle, but critical, difference between mounting using
          -v /mydata     (or Dockerfile VOLUME ["/mydata"])
          -v /mydata:/var/mydata

The first method allows Docker to control your data volume and Docker will automatically delete it once the last container referencing that volume is removed.  The second method cannot be used in a Dockerfile but has a significant advantage in that Docker will not delete the volume from disk until you explicitly call docker rm -v against the last container with a reference to the volume.

This "solution" isn't ideal (even if we use data only containers) because ultimately the data still resides on a single ephemeral host server and engineering a solution to support clustering across regions as well as managing regular backups will be inefficient and complex.

Update: distributed file systems are making good strides in this area.  See GlusterFSLizardFS, or powerstrip-flocker.  In theory, you would use a combination of a distributed file system for everyday live data handling and a traditional snapshot/archive offsite/cloud backup strategy for permanent storage.

2)  Mount a directory within the Docker container to a NAS or fuse it to a dedicated cloud storage solution (using something like s3fs, CloudFuse, or sshfs).  This approach is slightly less flexible because it's vendor specific, but overall the approach is simpler and more reliable.

Important: make sure to use local caching instead of fetching the file dynamically from across the network each time it's requested.  For example, s3fs has a use_cache feature that you must explicitly enable.

Conclusion: If cloud storage like S3 is designed for 99.99% availability and 99.999999999% durability and Netflix trusts them to store their content, isn't the answer obvious?  Use Option 2.  Update: due to network latency and potential unreliability, there are still performance (and single-point-of-failure) problems with Option 2 so I'm going to withhold judgement for the time being.


Trap #5

Installing or using SSH to login to your Docker container.


This is one of the most alluring and widespread traps for seasoned SysOps IT.  When it comes to Docker, though, you should use VOLUME or nsinit instead.  Jérôme Petazzoni (senior engineer at Docker) has a nice write-up explaining the reasoning and process.  Update: docker exec command now available in version 1.3.


P.S. Here's a nice read regarding Docker networking.

Audio Muse - volume 1


How to create a Drupal 7 newsletter (with HTML email formatting)


Enable Simplenews (and dependencies):
drush en -y simplenews mimemail mailsystem htmlmail 
drush updatedb 

Configure Simplenews to use HTML format and default send action is to "Send newsletter":

Configure your custom newsletter content type to use HTML format:

Give "Subscribe to newsletters" and "Choose to receive plaintext emails via HTML Mail" permissions to all roles and give the remaining Simplenews permissions to Content Administrators (or other desired role):

Additional secret-decoder-ring configuration:

Add newsletter subscribe block to desired theme section:

Logout of site and subscribe via block from previous step.  P.S. You can also auto-populate subscribers here:

Login to site and create newsletter:

Send newsletter to queue (by clicking the "Newsletter" tab of the newly created node):

Run cron to process the queue (may need to run multiple times if number of recipients is greater than "cron throttle" setting on /admin/config/services/simplenews/settings/mail):

P.S. It's better to use real cron instead of Drupal's default poormanscron.

Privacy under attack


I just finished reading No Place to Hide: Edward Snowden, the NSA, and the U.S. Surveillance State by Glenn Greenwald and was floored by the flagrant disregard and abuse of American and International privacy by the NSA and its affiliates:
"the government has the capability to activate cell phones and laptops remotely as eavesdropping devices.  Powering off the phone or laptop does not defeat the capability: only removing the battery does."  (p. 12)
"the NSA had what it called 'collection directly from the servers of these U.S. Service Providers: Microsoft, Yahoo, Google, Facebook, Paltalk, AOL, Skype, YouTube, Apple.'"  (p. 21)
"[FISA] ordered Verizon Business to turn over to the NSA 'all call detail records' for 'communications (i) between the United States and abroad; and (ii) wholly within the United States, including local telephone calls.'  That meant the NSA was secretly and indiscriminately collecting the telephone records of tens of millions of Americans, at least.  Virtually nobody had any idea that the Obama administration was doing any such thing. ... Soon after the story ran, the Associated Press confirmed from an unnamed senator what we had strongly suspected: that the bulk phone record collection program had been going on for years, and was directed at all major US telecom carriers, not just Verizon."  (p. 27, 72) 
"I realized that they were building a system whose goal was the elimination of all privacy, globally."  -Edward Snowden (p. 47)
"Obama's administration has prosecuted more government leakers under the Espionage Act of 1917 -- a total of seven -- than all previous administrations in US history combined: in fact, more than double that total."  (p. 50) 
"Through a carefully cultivated display of intimidation to anyone who contemplated a meaningful challenge, the government had striven to show people around the world that its power was constrained by neither law nor ethics, neither morality nor the Constitution."  (pp. 83-84)
Former NSA Director Keith B. Alexander's personal motto: "Collect it all"  (p. 95, 97, 169)
"the Washington Post reported in 2010 that 'every day, collection systems at the National Security Agency intercept and store 1.7 billion emails, phone calls, and other types of communications' from Americans."  (p. 99)
"the NSA has increasingly made use of a secret technology that enables it to enter and alter data in computers even if they are not connected to the Internet."  (p. 118)
"The [British] GCHQ has also conducted mass interception of communications data from the world's underwater fiber-optic cables.  Under the program name Tempora, ... the 'GCHQ and the NSA are consequently able to access and process vast quantities of communications between entirely innocent people'. The intercepted data encompass all forms of online activity, including 'recordings of phone calls, the content of email messages, entries on Facebook, and the history of any internet user's access to websites.'"  (p. 119)
"The NSA routinely receives -- or intercepts -- routers, servers, and other computer network devices being exported from the Unites States before they are delivered to the international customers.  The agency then implants backdoor surveillance tools, repackages the devices with a factory seal, and sends them on.  The NSA thus gains access to entire networks and all their users. ... Among other devices, the agency intercepts and tampers with routers and servers manufactured by Cisco to direct large amounts of Internet traffic back to the NSA's repositories. (There is no evidence in the documents that Cisco is aware of, or condoned, these interceptions.)"  (pp. 148-150)
"The key program used by the NSA to collect, curate, and search such data, introduced in 2007, is X-KEYSCORE, and it affords a radical leap in the scope of the agency's surveillance powers.  The NSA calls X-KEYSCORE its 'widest reaching' system for collecting electronic data ... the program captures 'nearly everything a typical user does on the internet,' including the text of emails, Google searches, and the names of websites visited.  X-KEYSCORE even allows 'real-time' monitoring of a person's online activities, enabling the NSA to observe emails and browsing activities as they happen."  (p. 153, see also 157, 159)
[2011-03-14 internal NSA memo] "BLARNEY began delivery of substantially improved and more complete Facebook content.  This is a major leap forward in NSA's ability to exploit Facebook using FISA and FAA authorities.  This effort was initiated in partnership with the FBI ... NSA is now able to access a broad range of Facebook data"  (p. 160)
[2011 presentation by GCHQ] "Many targets on Facebook lock down their profiles, so it is not possible to view all of their information... But passive offers the opportunity to collect this information by exploiting inherent weaknesses in Facebook's security model."  (p. 162)
[NSA presentation for a group of agency officials] "Put Money, National Interest, and Ego together, and now you're talking about shaping the world writ large."  (p. 167)
 "At the end of July 2013, the Pew Research Center released a poll showing that ... Americans now considered the danger of surveillance of greater concern than the danger of terrorism."  (p. 197)
"the argument that mass surveillance has prevented terror plots -- a claim made by President Obama and a range of national security figures -- has been proved false.  As the Washington Post noted in December 2013, in an article headlined 'Officials' Defenses of NSA Phone Program May Be Unraveling,' a federal judge declared the phone metadata collection program 'almost certainly' unconstitutional, in the process saying that the Justice Department failed to 'cite a single case in which analysis of the NSA's bulk metadata collection actually stopped an imminent terrorist attack.'"  (p. 202) 
"The usefulness of the bulk collection program has been greatly exaggerated.  We have yet to see any proof that it provides real, unique value in protecting national security." - Democratic senators Ron Wyden, Mark Udall, and Martin Heinrich -- all members of the Intelligence Committee, as quoted in the New York Times 
"The collect-it-all system did nothing to detect, let alone disrupt, the 2012 Boston Marathon bombing.  It did not detect the attempted Christmas-day bombing of a jetliner over Detroit, or the plan to blow up Times Square, or the plot to attack the New York City subway system -- all of which were stopped by alert bystanders or traditional police powers.  It certainly did nothing to stop the string of mass shootings from Aurora to Newtown.  Major international attacks from London to Mumbai to Madrid proceeded without detection, despite involving at least dozens of operatives."  (p. 203) 
"That [mass surveillance] capability at any time could be turned around on the American people and no American would have any privacy left, such is the capability to monitor everything -- telephone conversations, telegrams, it doesn't matter.  There would be no place to hide.  If this government ever became a tyrant ... the technological capacity that the intelligence community has given the government could enable it to impose total tyranny, and there would be no way to fight back because the most careful effort to combine together in resistance ... is within the reach of the government to know.  Such is the capacity of this technology."  - Frank Church, 1975
 "Not only is ubiquitous surveillance ineffective, it is extraordinarily costly. ... It breaks our technical systems, as the very protocols of the Internet become untrusted.  ... It's not just domestic abuse we have to worry about; it's the rest of the world, too.  The more we choose to eavesdrop on the Internet and other communications technologies, the less we are secure from eavesdropping by others.  Our choice isn't between a digital world where the NSA can eavesdrop and one where the NSA is prevented from eavesdropping; it's between a digital world that is vulnerable to all attackers, and one that is secure for all users."  - Bruce Schneier, as quoted in the Atlantic, January 2014
 "Democracy requires accountability and consent of the governed, which is only possible if citizens know what is being done in their name.  The presumption is that, with rare exception, they will know everything their political officials are doing, which is why they are called public servants, working in the public sector, in public service, for public agencies.  Conversely, the presumption is that the government, with rare exception, will not know anything that law-abiding citizens are doing.  That is why we are called private individuals, functioning in our private capacity.  Transparency is for those who carry out public duties and exercise public power.  Privacy is for everyone else."  (p. 209)

Looks like it's time to add _nomap to my SSID (still doesn't reign in Apple) , tighten my anonymity protection, and switch to MaidSafe...

Black screen on Google Maps


A recent update to Google Maps caused my screen to black out with a WebGL error:

Fortunately, the solution was pretty easy: load Google Maps in Lite Mode   (source)

Recommended apps for rooted Android phones


I recently rooted and updated my Droid Razr M to Cyanogenmod 11 and I'm loving it!  Now that I have the Power of Grayskull, what can I do with it?

  1. Install Adblock Plus for Android (not even Google can stop me now!)
  2. Encrypt my text messages with WhisperPush
  3. Enable Privacy Guard by default for newly-installed apps and apply it to any desired existing apps
  4. Install System app remover pro, set Menu > Settings > Remove Ads, then uninstall all the Amazon/Facebook/NFL/etc. bloatware on my phone!  Also remove (or disable) Google+, Hangouts, Movie Studio, News and Weather, Play Books, Play Games, Play Movies & TV, Google Music, and Play Newsstand
  5. Use System app remover pro to uninstall the "Google Keyboard" system app -- NOT the "Android Keyboard (AOSP)" -- then install the Android L Keyboard.  After installing via Recovery, go into Settings > Language & input and make sure "Google Keyboard" is checked and set as Default and then click on the settings icon for "Google Keyboard" and then click on "Advanced settings" and then click on "Color scheme" to change it to "Material".
  6. Enable Advanced Reboot (additional options for Recovery and Bootloader when selecting phone reboot)
  7. In the Google Settings app, select the Ads option and check Opt out of interest-based ads, then touch the Reset advertising ID option
  8. In the Play Store app, press the three-bar icon in the top left and select Settings -- make sure apps auto-update over Wi-Fi only (or disable auto-update if you wish) and also uncheck Add icon to Home screen for new apps.  I also personally like to require a password for all purchases through Google Play.
  9. In Settings > Lock screen > Screen security > Screen lock select a secure method for securing your phone.  I also like the Quick unlock feature with a 5 second automatic lock (instead of instant power lock).
  10. Enable the DSP Equalizer for better sound playback
  11. If using Verizon, call your phone and leave a voicemail.  Then when you tap the notification to call your voicemail and are prompted for the number use *86
  12. Set up your wallpaper by long pressing the background
  13. Also, set up your widgets by long pressing the background:
    • cLock: show AM/PM indicator, use OpenWeatherMap (30 minute update interval with VClouds icon set), and uncheck calendar events; then press the checkmark in the upper right, press and hold the widget until "Remove" appears then let go and resize the widget to use the full upper half of the home screen.  Note: if you need to update these settings later, you need to go into Settings > Lock screen > Clock widget
    • Torch (useful flashlight widget)
    • remove the Google Play 4x2 widget on the second screen by pressing and holding it and then dragging it to "Remove"
  14. Install Deep Sleep Battery Saver
  15. Install Pandora by Holonation
  16. Install Grooveshark by Holonation
  17. Install YouTube
  18. Install Xposed Framework and then download and enable the following modules:
  19. Install App Ops shortcut (nice try, Google)
  20. Install Rooted AnyConnect VPN client by Cisco and then install Greenify to hibernate the app so it only runs when you need it (also enable the Greenify Xposed module that it comes packaged with)
  21. Install CCleaner (but deny access to call log, SMS messages, etc. when prompted by Privacy Guard)
  22. Install Nandroid Manager, Online Nandroid Backup, and unlock key and then set up your daily full device backup
  23. Install Startup Manager (uncheck Show System Boot Time in settings) and disable any app that doesn't need to start up automatically
  24. Install SELinux Mode Changer and change the mode to "Enforcing"
  25. Install Secure Update Scanner
  26. Install SecDroid and then run it and press the "Secure Phone" button to enable it.  Then add it to Startup Manager (Customize tab) to remind you to enable it after every boot.  Note: you can verify it's working by re-running the Secure Update Scanner tool from the previous step and it won't be able to complete a scan.
  27. Install DroidWall then use 3-dot menu in bottom-right corner to enable firewall and also apply changes (p.s. allow "Downloads, Media Storage, Download Manager" to enable Play Store installs. p.p.s. make sure to disable firewall before uninstalling if you ever need to remove it entirely)

Other Recommended Apps:

...And for the truly paranoid: https://blog.torproject.org/blog/mission-impossible-hardening-android-security-and-privacy

doubleTwist on Linux


Update: a much easier solution was to use the Shuttle Music Player instead.  It automatically downloads album art, includes an equalizer, integrates with RunKeeper, and supports gapless playback and volume normalization for free.

Update #2: the latest Shuttle update ruined the app (removed features, themes, etc.) so for now I'm just using Apollo.

My old Shuttle settings (for reference):


  • Default Page: Albums
  • View Options > Albums: Grid
  • Enable Lockscreen Controls: checked
  • Hybrid
  • Download Artwork Automatically: checked
  • Download Via Wi-Fi only: checked
  • Use Gmail Style Placeholders: unchecked
  • Pause on Disconnect: checked
  • Gapless Playback: checked
  • Enable Scrobbling: unchecked


Legacy (non-working) instructions for reference:

There are a lot of online posts indicating doubleTwist does not work on Linux (or it does with with MTP but is a pain to set up).  For me, though, doubleTwist works fine with Lubuntu and CyanogenMod 11.

When I plug in my device via USB, it's automatically detected and file manager opens it via the MTP protocol:

When I add mp3 files to "SD card/Music", "SD card/doubleTwist", or "Internal storage/Music" doubleTwist loads them just fine.  I can also delete files in the "SD card/doubleTwist" folder via the doubleTwist app GUI.

The only thing that's annoying is that many of my albums do not display a cover art image.  I can either pay $$ for doubleTwist Pro (which automatically retrieves missing high-quality album art) or I can add the artwork manually to each song's ID3 metadata.  Note: This needs to be done on your PC prior to copying over the files to the doubleTwist folder (apps like Cover Art Downloader do not work).

Since doubleTwist doesn't support custom ROMs (like CyanogenMod), I decided on the latter approach using a handy (albeit non-intuitive) app called MP3 Diags.

1. sudo apt-get install mp3diags

2. Open the app via Start > Sound & Video > MP3 Diags and click on the sprocket icon, then check all the folders that have music and click the Scan button

3. Click on the gray disc icon and then select the desired album you want to add cover art to by clicking on the blue right arrow.

4. The selected album songs will now appear in the list.  To perform a general cleanup, click on the hammer #4 icon and then click Yes.  Repeat this process for each desired album by clicking the blue right arrow and repeating this step.

5. Next, click on the tag icon

6. Click the MusicBrainz icon, then check the box for "Track count", and click the Search button.  After verifying the information looks correct, click the "Save all" button and then click the floppy disk icon to apply the changes.  Repeat this process for each album by clicking the blue right arrow and repeating this step.

7. Once done editing the tags, close the Tag Editor screen.  Make sure the gray disc icon is selected and then click on the wave-graph icon then click "Normalize" to normalize the sound levels.  Repeat this process for each album by clicking the blue right arrow and repeating this step.

8. Next, click on the pen-on-blue-folder icon

9. Click on the "%n.%t" textbox icon and in the popup create your file rename format (I used %n - %t) and then click OK.

10. Click on the hammer icon to rename the files (click Yes to confirm).  Repeat this process for each album by clicking the blue right arrow and repeating this step.

11. Close the MP3 Diags application and copy your updated music folders to your smartphone.  Open doubleTwist to verify the album art appears and the tracks play correctly.

Tutorial: Installing CyanogenMod on Verizon Droid Razr M KitKat (4.4.2)


Update: CM 11 still has microphone, audio playback, and WiFi bugs so with dhacker29 leaving the CM team, I've switched to his ROM, Gummy, for improved stability.  I've updated the original guide below accordingly:

Update #2: The speakerphone issue has been fixed and WiFi issue has been fixed on both CyanogenMod and Gummy.  Also, since Motorola support appears to remain strong on CyanogenMod, and it includes Privacy Guard (not found on Gummy), and the Gummy ROM OTA update process is currently broken, it's probably best to stick with CyanogenMod for now.  I've reverted the original guide accordingly:

Update #3: There continues to be WiFi and GPS issues using the KitKat bootloader so you'll need to downgrade to the JellyBean bootloader and install the _jbbl version of the nightlies instead (for example, see bug MC-39).  I've updated the guide accordingly

Update #4: The latest Verizon OTA update (183.46.15) patched the rooting capability of the towelroot exploit described below.  If you're starting with a stock Razr device, this guide won't help.  If your bootloader is currently unlocked, see this guide for regaining root access.

Update #4b: Some users are reporting success with kingroot for rooting the device but still no word on a reliable method to unlock the bootloader...

Update #5: This somewhat outdated guide is useful for switching from the Verizon stock OS to Cyanogenmod but if you already have a rooted phone or are already running Cyanogenmod and you want to upgrade to version 12.1, use this guide (the key is to use TWRP, install SuperSU when prompted in the recovery reboot process, and to use TK GAPPS [I like modular micro]).  Once installed, make sure to enable root and reset GPS.  Also, here are the Xposed instructions if you need them (get the latest xposed-v*-sdk22-arm.zip and XposedInstaller_3.0-alpha*.apk files here).  And don't forget to choose a cool theme (I like Euphoria Dark)


So you've heard about this great utopia of milk and honey where your smartphone runs faster, uses less battery life, gives you more control, and provides more frequent updates and you want to try it out yourself.  Awesome.  But before we dive into that adventure, you must swear with an oath to obey the:

~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~

Ten Commandments of ROM Flashing

  1. Thou shalt have no other ROMs before CyanogenMod LineageOS
  2. Thou shalt not proceed if thy exact device and Android version is not listed at sbf.droid-developers.org
  3. If thou knowest not what thou art doing, thou shalt not do it
  4. Remember thy backups, to keep them holy -- if any warning screen says "CANNOT BE UNDONE", cancel, create full (nandroid) backup, then continue with action.
  5. Thou shalt charge thy smartphone completely (100%) before attempting a ROM flash
  6. Thou shalt not kill thy smartphone by skipping steps, ignoring warnings, tl;dr, voodoo programming, shotgun surgery, panicking, etc.
  7. Thou shalt not commit adultery by installing zip, img, bin, etc. files from another manufacturer, device, version, or random outdated forum post onto thy device
  8. Thou shalt use the manufacturer's official USB cable (OEM)
  9. If thy device is Motorola, thou shalt use the Motorola version of fastboot
  10. Thou shalt give praise and thanksgiving to the many brilliant hackers that have made all of this possible!
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~

Now that we've established the ground rules, it's important to understand the technical terminology.  Read all of this post before moving on with the guide below (see Commandment #3 and #6).

Regarding Commandment #7, the following device terms are equivalent for the purposes of this guide:
  • Smartphone model: "xt907", "Droid Razr M", "Razr M", and "msm8960" (Note: the "Razr HD" device is similar enough to the "Razr M" that you're generally okay with files for that device as well)
  • Android version: "KitKat", "KK", "4.4.2"
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~


Okay, if you've made it this far you're pretty committed to doing this.  So, without further ado:

A. Gain root access (similar to "jailbreak" on iOS devices)

1. On your phone, go to Settings > Security & Screen Lock
2. Check Unknown sources and (if exists) uncheck Verify apps (click OK on any security prompts)

3. If Settings > Developer options doesn't exist, go to Settings > About Phone and scroll down to the bottom.  Tap Build number seven times to enable developer mode

4. Go to Settings > Developer options, enable it if disabled, and check USB debugging

5. Open a web browser and download towelroot: https://tinyurl.com/khf62jr  [mirror]  {credit: geohot}
6. Click OK on the security warning to download.
7. Go to your phone apps and click on Downloads
8. Click on tr.apk and click Install and then Open
9. Click on the make it ra1n button.  You should now have root access!

10. Open a web browser and download Root Checker: https://tinyurl.com/ppg9cj7  {credit: CMDann}
11. Click the Verify Root Access button and make sure a green checkmark appears

12. On your phone, go to Settings > Location and turn Location ON (say Yes to any prompts that appear)
13. Open a web browser and download GPS Test: https://tinyurl.com/puwhtue  {credit: Chartcross Limited} and wait until "Accuracy (feet)" is greater than 0 (may take a few minutes; faster if you're outside with a clear view of the sky).  Note: this is NOT an optional step.  Activating GPS prior to flashing the ROM is required (see explanation here).
14. Congratulations, you're ready for the next phase!

B. Unlock the Bootloader

1. Open a web browser and download Moto-pocalypse: http://goo.gl/ioGJDo  [mirror]  {credit: djrbliss}
2. Click OK on the security warning to download.
3. Go to your phone apps and click on Downloads
4. Click on Motopocalypse.apk and click Install and then Open
8. Click on the Unlock Bootloader button.

9. Hold your power button and select Power off to turn off your phone.
10. Wait a few seconds for your phone to restart (if it doesn't, hold your power button down until it starts).  Your phone should display this message:

11. Congratulations, you're ready for the next phase!

C. Install device drivers

1. On your PC, download the latest Motorola drivers:

2. Connect your phone using the manufacturer cable (see Commandment #8) and make sure the device is recognized and you can browse the phone contents.
3. Congratulations, you're ready for the next phase!

D. Install custom Recovery image

1. Hold down power, volume up, and volume down buttons simultaneously.  When the device powers off, let go of the power button but keep holding down the volume up and volume down buttons until the bootloader screen appears.
2. Quickly press the volume down button until AP Fastboot is highlighted and then press the volume up key.
3. The screen should now be in fastboot mode and should say Device is UNLOCKED. Status Code:3.  At the bottom of the screen it should say usb connected

4. On your PC, download the Motorola version of fastboot (see Commandment #9) {credit: mattlgroff}
5. Extract the fastboot file for your operating system to somewhere in your PATH.  For example, on Linux:
  • chmod +x fastboot
  • sudo mv fastboot /usr/bin/
6. Open a terminal command prompt and type fastboot devices
7. You should see something like   {SERIAL NUMBER}   fastboot
8. On your PC, download the Razr HD/M Recovery image: http://goo.gl/keFZNy  [mirror]  {credit: dhacker29}  Update: some individuals have reported issues with installing the latest version of CM11 using the Clockworkmod (CWM) recovery mentioned in this step.  If you have trouble, try using the Team Win (TWRP) recovery image (TWRP- instead: http://d-h.st/nw4  {credit: dhacker29}
9. In a terminal command prompt, browse to where the image file is located and then type fastboot flash recovery CWM6049-RAZR-HD_M-KITKAT.img
10. Hold down power, volume up, and volume down buttons simultaneously.  When the device powers off, let go of the power button but keep holding down the volume up and volume down buttons until the bootloader screen appears.
11. Quickly press the volume down button until Recovery is highlighted and then press the volume up key.
12. Your phone will boot into recovery mode:

13. Congratulations, you're ready for the next phase!

E. Create full device backup (a.k.a. "nandroid")

1. Use your volume down (or volume up) button to highlight backup and restore then press your power button to select it
2. Use your volume down button to highlight choose default backup format then press your power button to select it
3. Use your volume down button to highlight tar + gzip then press your power button to select it
4. Use your volume down button to highlight backup to /storage/sdcard1 then press your power button to select it
5. A full backup of your device will be created and stored on your SD card
6. Use your volume down button to highlight +++Go Back+++ then press your power button to select it
7. Use your volume down button to highlight mounts and storage then press your power button to select it
8. Use your volume down button to highlight mount USB storage then press your power button to select it
9. Your PC should now detect your phone.  Open the clockworkmod/backup/ folder.  Move the backup folder you just created (date timestamp may be off but that's okay) to your PC for safekeeping.
10. Use your volume down button to highlight +++Go Back+++ then press your power button to select it
11. Unplug the USB cable from your PC
12. Press the power button and let go.  The phone will restart.
13. Congratulations, you're ready for the next phase!

F. Factory reset

1. Now that you have a full device backup, it's time to perform a factory reset of your phone.  This will ensure any previously encrypted partitions are deleted, permissions are reset to factory defaults, and no previous phone settings or installed applications will interfere with the CyanogenMod install.
2. Follow these instructions to reboot into safe mode and then follow these instructions to reset your phone to factory settings (note: if the phone goes into a bootloop, just hold down the power button and both volume buttons simultaneously until the screen turns black and then release the power button while still holding the volume buttons and then when the boot menu appears, use the Volume Down button to highlight "Factory" and press the Volume Up button to select it.  Once the phone loads normally, turn it off and then back on again.) 
3. Follow these instructions to reformat your SD card
4. Due to this bug and other various GPS and WiFi bugs with the KitKat bootloader, follow these instructions to downgrade your phone to Jelly Bean (4.1.2) ...P.S. it's just the bootloader that will be running JellyBean, your actual phone OS will eventually be running KitKat (4.4.4) once you complete this guide
5. Re-do Section A above (gain root access)
6. Congratulations, you're ready for the next phase!

G. Install CyanogenMod

1. Connect your phone to your PC via the USB cable and browse to the SD card
2. On your PC, download the latest CyanogenMod snapshot with JellyBean bootloader
3. On your PC, download the desired Google Apps 4.4.4 image (I prefer the "Micro Modular" package)
4. On your PC, download the stock Motorola boot logo image: http://goo.gl/YYmG2f  [mirror]  {credit: aviwdoowks}
5. Copy the zip files from steps 2-4 to your phone's SD card
6. Unplug the USB cable from your PC
7. Hold down power, volume up, and volume buttons down simultaneously.  When the device powers off, let go of the power button but keep holding down the volume up and volume down buttons until the bootloader screen appears.
8. Quickly press the volume down button until Recovery is highlighted and then press the volume up key.
9. Once your phone boots into recovery mode, use your volume down button to highlight wipe data/factory reset then press your power button to select it
10. Use your volume down button to highlight Yes - Wipe all user data then press your power button to select it
11. Use your volume down button to highlight advanced then press your power button to select it
12. Use your volume down button to highlight wipe dalvik cache then press your power button to select it
13. Use your volume down button to highlight Yes - Wipe Dalvik Cache then press your power button to select it
14. Use your volume down button to highlight +++Go Back+++ then press your power button to select it
15. Use your volume down button to highlight install zip then press your power button to select it
16. Use your volume down button to highlight choose zip from /storage/sdcard1 then press your power button to select it
17. Use your volume down button to highlight stock.boot.logo.flashable.zip then press your power button to select it
18. Use your volume down button to highlight Yes - install ... then press your power button to select it
19. Use your volume down button to highlight choose zip from /storage/sdcard1 then press your power button to select it
20. Use your volume down button to highlight cm-11-2014... then press your power button to select it
21. Use your volume down button to highlight Yes - install ... then press your power button to select it
22. Use your volume down button to highlight choose zip from /storage/sdcard1 then press your power button to select it
23. Use your volume down button to highlight pa_gapps-modular... then press your power button to select it
24. Use your volume down button to highlight Yes - install ... then press your power button to select it
25. Use your volume down button to highlight +++Go Back+++ then press your power button to select it
26. Make sure reboot system now is highlighted, then press your power button to select it
27. Your phone will reboot, the Motorola logo will now display (instead of the annoying "Bootloader Unlocked" nag screen), and CyanogenMod will begin loading.  Be patient -- may take up to 10 minutes to load for the first time.

Congratulations, you've installed CyanogenMod on your Razr M!

Note: If CyanogenMod doesn't load after 10 minutes or you run into any other problems (e.g. bootloop, etc.), you can always start over by re-flashing your device back to factory settings.

In a follow-up post I'll describe some recommended settings and apps.

Tutorial: Restore Verizon Droid Razr M KitKat (4.4.2) firmware


If you've done something to your Droid Razr M (version 4.4.2 - KitKat) and now it won't boot and you've tried everything else to fix the problem and are just about to toss it in the trash as a bricked device, try re-installing the factory default firmware:

1. On your PC, install the latest Motorola drivers and reboot

2. Connect your phone to your PC using the USB cable and make sure your phone battery is charged 100% (if not, wait until fully charged)

3. On your phone, hold down power, volume up, and volume down buttons simultaneously.  When the device powers off, let go of the power button but keep holding down the volume up and volume down buttons until the bootloader screen appears.

4. Quickly press the volume down button until AP Fastboot is highlighted and then press the volume up key.

5. The screen should now be in fastboot mode and should say "Battery OK" and "usb connected"  (note: if it says "Battery low", use these instructions to charge your phone)

6. On your PC, download the Motorola version of fastboot  {credit: mattlgroff}

7. Extract the fastboot file for your operating system to somewhere in your PATH.  For example, on Linux:
  • chmod +x fastboot
  • sudo mv fastboot /usr/bin/
8. Open a terminal command prompt and type fastboot devices

9. You should see something like  {SERIAL NUMBER}   fastboot

10. On your PC, download the appropriate Android firmware:
11. Extract the zip file contents to a temporary location on your PC

12. In a terminal command prompt, browse to where the extracted files are located and then type the following commands (one at a time):

fastboot -w
KitKat (4.4) ONLY: fastboot flash partition gpt.bin
fastboot flash sbl1 sbl1.mbn
fastboot flash sbl2 sbl2.mbn
fastboot flash sbl3 sbl3.mbn
fastboot flash rpm rpm.mbn
KitKat (4.4) ONLY: fastboot flash tz tz.mbn
fastboot flash aboot emmc_appsboot.mbn
fastboot flash modem NON-HLOS.bin
fastboot flash fsg fsg.mbn
fastboot erase modemst1
fastboot erase modemst2
fastboot flash logo logo.bin
fastboot flash boot boot.img
fastboot erase tombstones
fastboot flash recovery recovery.img
fastboot flash system system.img
fastboot continue

Note: initial boot may take up to 10 minutes to complete

OK Go having fun with optical illusions


Google gets snarky with app permissions



I was trying to find an exercise app the other day and eventually decided on one that required minimal permissions -- in particular, full Internet access wasn't listed.  After installing the app, I was surprised to see ads.  I dug a little deeper and found this snarky update from Google:

Note: These days, apps typically access the Internet, so network communication permissions including the “full Internet access” permission have been moved out of the primary permissions screen.
What?!  Are you serious?!  One of the most potentially dangerous permissions is no longer even listed when installing Android apps?  It's like Google is giving a green light to every malware/backdoor/stalker/advertiser slimeball developer.  Lame!

I mean seriously, if I wanted NSA to know my every move I'd just install the Moves app or wear those disturbing fitness wristbands and be done with it.

An additional legalese clause is even more sinister:
Once you’ve allowed an app to access a permissions group, the app may use any of the individual permissions that are part of that group.
When an app updates, it may need to use additional capabilities or information controlled by permissions.
If you have automatic updates enabled, you won't need to review or accept these permissions as long as they are included in a permissions group you already accepted for that app.

In other words, if I allow full Internet access (that's now stupidly grouped in the generic Other group) for an app that legitimately needs it, I've now implicitly given the app owner permission to write to my Google+ account (among many "other" vague things):

Double lame! (and misleading, unethical, and possibly illegal)

Google, you can keep your "simplified permissions" -- give me back my privacy control!


In addition to already disabling auto-update, it looks like I'll need to manually check the full permissions from now on.