Delete duplicates from over 150,000 bookmarks on Google Chrome

Question

I believe I tried all the bookmark management apps on the Chrome web store, deleted part of the bookmarks manually (yes, I even attempted that), organize them within folders and all, but they just came back every time I launched Chrome.

If you're wondering how I ended up with 150,000 bookmarks I'm not sure either. All I know is that they'd been imported from Firefox at first, with StumbleUpon bookmarks and all, and they just seem to have duplicated themselves with every sync since maybe... 2009. I've got a lot of empty folders in the mix as well, and they seem to increase in number too each time I take a look at my bookmarks.

Right now I'm running AM-DeadLink, who has been trying to delete the duplicates for maybe 10 hours now, and I'm not sure if it's even working any more since the Bookmarks file size in AppData/Local/Google/Chrome hasn't dropped from 51mb.

I don't want to delete all my bookmarks since I have important stuff in there. What I need is to get rid of all those duplicates for good. Any ideas?

---

Edit: So, a year later, the problem remains, and now I have another question:

Right now there's a total of 151739 bookmarks in my account as Chrome Sync indicates, but the categories under Chrome Sync are not individually editable or removable. The only editing option I get is to delete the sync data as a whole.

I would like to delete only the bookmarks and leave the rest for now until I figure out what I want to keep. How do I delete the bookmarks without touching anything else? Is that possible?

Answer 1

I'm in the process of dealing with this myself. I had 260,000 bookmarks, the vast majority of which were "phantoms" -- folders with no name, many duplicates, etc. Every time I started Chrome, it would consume many gigabytes of RAM, to the point of really impacting my productivity.

I decided to delete all bad bookmarks, but I had to wipe out the cloud copy of my Chrome sync data entirely to get the change to "stick". I don't have a quick solution, but I believe the following works.

I started with a computer that had a complete copy of my Chrome data. I backed up the profile directory with all that data. (See Where are the user profile directories of Google Chrome located in?).

I started Chrome, waited for it to settle, went to settings in Chrome and turned off sync. Then I went to https://www.google.com/settings/chrome/sync and clicked "Stop and Clear", which disables sync and deletes all your Chrome profile data (including all the duplicate and phantom bookmarks) from the Google Cloud, but it should still be stored your Chrome profile on this computer.

I used the bookmark manager to manually delete all the phantom bookmarks. Luckily, most of mine were organized into duplicate folders so I only had a dozen or so things to delete. It still took a long time. The mass of phantom bookmarks brought Chrome to a crawl -- I'd right click on one of these duplicate folders and it was sometimes minutes before the menu with the "delete" option appeared.

So after getting rid of all those bookmarks on that one machine, I exited Chrome just to give it a chance to recover. I restarted Chrome, went into settings and turned sync back on. It uploaded the remaining bookmarks plus the passwords etc. that are still saved on that computer.

Now on each other computer, I exited Chrome, moved my Chrome profile data to the trash (because those copies of the profile still have all the phantom bookmarks), restarted Chrome, signed in, and just waited until sync could restore all my information.

FYI: I've been looking all over for a way to force Chrome to sync everything right this minute. I've found plenty of reasonable suggestions, but so far none of them work. It sometimes takes minutes or hours before the sync is complete, go figure.

Answer 2

Having tested all of the suggestions, it seems using bookmark manager to manually delete all the duplicate bookmarks is the most reliable (same behaviour and resolution as detailed in Garrett Mitchener's response above.)

The main sticking point was to ensure only the duplicates were deleted. In other words, get a list of unique bookmarks in the bookmark manager, to compare after the clean-up.

This worked quite well using standard Linux tools on Ubuntu Trusty:

Back up the bookmarks file in case a unique folder accidentally gets deleted:

$ cp -av .config/google-chrome/Default/Bookmarks{,.orig} ‘.config/google-chrome/Default/Bookmarks’ -> ‘.config/google-chrome/Default/Bookmarks.orig’

Get a count of all URLs:

$ grep -c '"url": ' .config/google-chrome/Default/Bookmarks

Get a count of all unique URLs:

$ grep '"url": ' .config/google-chrome/Default/Bookmarks | awk '{print $2}' | sort | uniq | wc -l

Piping grep into awk is a great deal faster than awk matching alone, and awk has to be piped into sort in order to accurately get unique entries.

Stick them all in a file, may as well trim off the extraneous double-quotes while we're at it:

$ grep '"url": ' .config/google-chrome/Default/Bookmarks | awk '{print $2}' | sort | uniq | sed 's/^"//;s/"$//' > Bookmarks-Original.txt

Perform the clean-up in the Bookmark manager, then extract all unique URLs from the bookmarks file:

$ grep '"url": ' .config/google-chrome/Default/Bookmarks | awk '{print $2}' | sort | uniq | sed 's/^"//;s/"$//' > Bookmarks-New.txt

Run the comparison:

$ for URL in $(cat Bookmarks-Original.txt); do grep -q $URL Bookmarks-New.txt || echo $URL; done > Bookmarks-Discrep.txt

Now it's possible to search the original Bookmarks file, extract the metadata for the original and carefully add back to the new Bookmarks file (taking a backup of the newest file first), e.g.

{
            "date_added": "13026268601621410",
            [...]
            "url": "https://wiki.mozilla.org/Security/Server_Side_TLS"
         },

If the metadata are unimportant, it's easier just to create new bookmarks for each in the Bookmark manager and move into the relevant folder.

Answer 3

I shut off bookmarks sync on Icloud and deleted the icloud extension in chrome. the problem stopped. not a big deal I dont use safari on my iphone but use chrome as my default browser so I still get synced bookmarks.

Answer 4

I tried some of the solutions provided without success. I then ran across this Reset Sync solution (from "Johnny") on the Google Product Forum that was simple and worked. You remove the bookmarks from all of your devices except the one that has your desired bookmarks and then Reset Sync.

As was pointed out in another thread you may need to uninstall/reinstall Chrome on an Android device to get rid of the bookmarks.

Answer 5

I love Google Chrome but was about to the point of changing to another browser because of constant multiplication of bookmarks. I decided, however, that I might be complicating things by trying to correct the problem on multiple devices and not letting the sync feature do its job. The more changes I made, the worse things got. Listed below are the steps I took; and after 48 hours, I have no more duplicates and everything stays in sync on all five devices.

YOU MUST GET RID OF ALL BOOKMARKS ON EVERY DEVICE AND THEN START A NEW SET OF BOOKMARKS.

• Disable the Sync feature on EVERY device on which your Google account was used. Google warns us to not open Chrome on public computers because the program remains resident on those computers. That could be a great problem because how can you correct what was already done? I decided to hope for the best.
• Disconnect from your Google account on EVERY one of those devices. You can review your connected devices at https://myaccount.google.com/u/1/device-activity
• On a PC, Bookmarks Manager (Ctrl+Shift+O) is an easy way to remove unwanted duplicates
• On ONE device, make a good backup of the bookmarks you want to keep. Ctrl+Shift+O will get you to the Bookmark Manager. In the blue bar, select ORGANIZE, then "Export bookmarks to HTML file...", and save the resulting file in a place you can find it later.
• DO NOT SIGN IN on any device for now.
• TOTALLY REMOVE all bookmarks from every device. Bookmarks Manager works great for a PC; but on some devices, it may be necessary to uninstall Google Chrome and then re-install it. Make sure you have NO bookmarks on any device.
• Sign in to your Google account on ONE device (I preferred to do this on a PC).
• Import the file with good bookmarks that you saved. Use Ctrl+Shift+O, Organize, "Import bookmarks from HTML file..."
• Turn on the Sync feature on only this computer. I left my computer turned on for several hours to make sure everything got to the Google server.
• Sign in to you Google account on each device and activate the Sync feature.
• You should be set to go!!!
• Just don't go wild making so many bookmark changes on every device.

I hope this has been a help.

If these instructions are really accurate, it would be nice if Google could incorporate something similar, but in better format, for their millions of users because it sounds like it is a very wide-spread problem.