99 Lines of Code to Fix Onetab's Crashes & Back Up Raw Data

Foreword#

I learned about Onetab almost at the same time as Chrome, back in university; but as a student, web surfing wasn't much of a burden, so I used it rarely.
After starting work, opening 100 tabs in the browser became routine, and supervisors/colleagues frequently asked for materials. At that point, Onetab, as a web page notebook, combined with Ctrl+f to search keywords, was simple and easy to use.

Why Not Use Bookmarks?#

The end result of my use of bookmarks was that, when the bookmarks bar expanded, it covered the entire screen.

Why Not Use History?#

History still contains other irrelevant information, and Chrome's history is not reliable.

Why Not Use Other Knowledge Management Tools?#

1. Note-taking apps are too heavy
2. For Onetab-like apps, due to my ingrained usage habits, I couldn't find a replacement that was lightweight enough and more efficient
3. Onetab added dark mode, and caching previous/next/current tabs is truly practical

Situation#

Five days ago, after a Windows update restart, I reopened Edge, and Onetab stayed in a loading state. After a while, the page crashed and reported an Out of Memory error.

• I have used Onetab continuously since the new Edge launched in early 2020, with 60,000 web pages recorded, and I have never reinstalled the system.

The Three-Pronged Attempt#

1. Close all web pages and extensions, restart Edge, and reopen Onetab.
   Failed, reporting an Out of Memory error.
2. Restart the computer, restart Edge with no background processes, and reopen Onetab.
   Failed, reporting an Out of Memory error.
3. Fully copy the “C:\User” folder (the Windows user data directory) and restore it to a PC with a fresh Windows 10 installation.
   Failed, reporting an Out of Memory error.

At this point, it was basically certain that the extension crash was caused by too much data.

• In fact, even with 50,000 records, opening it was already very slow and took over 30s.
• I just didn't expect that the memory-hungry Chrome would also put a cap on the memory usage of its underlings.

Target#

By now the problem had been found. Fortunately, the data was not lost, although its integrity was unknown.
The data accumulated over more than two years is also used very frequently in daily work, so it must be recovered.

1. Recover Onetab data as much as possible and export it in a data format readable by both machines and humans
2. On the basis of 1., implement an automated tool for fully backing up Onetab
   Onetab's built-in export is a plain txt file, still requires manual copying, and lacks the field for the tab group's creation time
3. Implement an Onetab-compatible app, enhance data search functionality, and make it compatible with the data format exported in 1. and Onetab's original data format

Actions#

Now that the cause and goals were identified, implementation was no longer aimless.

1. Googling Prior Experience#

Search for “onetab data location”
I quickly found where Onetab—or rather, Chromium-based browser extensions—stores its data

Then I searched for “onetab data export” and found another predecessor's open-source solution, based on Go.

I originally thought it would end there and I'd take the copy-paste route, but an issue record said it no longer worked. I also had no Go debugging experience. After looking at the code, which was under 100 lines, I decided to implement it myself.

2. Get to Know leveldb#

• A NoSQL database open-sourced by Google
• Single-threaded, high-performance, local instances only
• Uses a lot of memory for acceleration during operation

A Standard leveldb Database Instance Is in the Form of a Folder#

• LOCK database lock
• MANIFEST header file
• *.ldb main data file

From the introduction to the open-source solution, I learned that Onetab uses leveldb to store user data.
I have also developed Chrome extensions myself, and I remember that localization could only use the two APIs localStorage and IndexedDB, because extensions running in the browser do not have capabilities such as file I/O and TCP connections.
localStorage has a maximum of 5M, so 50,000 relatively long JSON records could not possibly fit. Onetab should have been using IndexedDB to store data. What was this leveldb about, and its contributor was also Google?
After some searching, I learned that leveldb is the underlying implementation of IndexedDB; IndexedDB is actually the upper-layer API (a misleading-name series).

Google Chrome stores some data in Local Storage in SQLite format (.localstorage files) in a folder.
Some other data stored in the IndexedDB folder (for each profile) uses the LevelDB format. It is an open-source key-value storage format developed by Google and hosted on GitHub.
Modifying LevelDB-format files outside Chrome is not a simple process, because you need to implement a compatible comparator to inspect Chrome's Indexed DB leveldb instance.

3. leveldb GUI Tools#

After understanding the whole story, in order to first confirm (take a shortcut) the data situation, I looked for leveldb GUI tools.
Mainstream tools like Navicat and DataGrip certainly don't support it, so I continued surfing GitHub.

An Electron GUI Implementation, Compilation Failed#

Another Electron GUI Implementation, Could Not Read Data#

A C++/Qt GUI Implementation, Successfully Read the Data#

And I could clearly see the key names used by Onetab.

• Note: This software is open-source under GPL v3, but the binary version requires payment; new users can try it. When I was writing this article, the official website was down and registration was impossible, so you can download an old version and run it without registration or login.

4. Code Implementation Based on node.js 16.15#

Although the two Electron GUI implementations ended in failure, I did discover the level package they depended on, which provides a very convenient out-of-the-box API.

• At first I wanted to implement this in Python, but Python's level dependency needs to be compiled and installed under Linux, so I gave up on that.

$ npm install level
$ npm install mongodb

const {promises: {writeFile, mkdir}} = requir('fs');
const {Level} = requir('level')
const MongoClient = requir('mongodb').MongoClient
// halo有点bug, 出现require会崩溃

const USER_HOME = (process.env.HOME || process.env.USERPROFILE).replaceAll("\\", "/")

const chrome = {
    name: "Google/Chrome",
    uuid: "chphlpgkkbolifaimnlloiipkdnihall"
}
const edge = {
    name: "Microsoft/Edge", // Edge || Edge Beta || Edge Dev || Edge Canary
    uuid: "hoimpamkkoehapgenciaoajfkfkpgfop"
}
// const firefox = "" // 火狐的扩展和目录结构与以上两位好兄弟不太一样, 无使用经验, 请自行修改

const winPath = `${USER_HOME}/AppData/Local/${edge.name}/User Data/Default/Local Extension Settings/${edge.uuid}`
// 以下两条为 Copilot 智能添加的代码, 请自行核对路径是否正确
// const macPath = `${USER_HOME}/Library/Application Support/${edge.name}/User Data/Default/Local Extension Settings/${edge.uuid}`
// const linuxPath = `${USER_HOME}/AppData/Local/${edge.name}/User Data/Default/Local Extension Settings/${edge.uuid}`

const ldb = new Level(winPath)
const url = "mongodb://alpha:123456aB@localhost:27017/?authSource=admin"

// 异步读取onetab的leveldb中的数据
const getOnetabData = async () => {
    await ldb.open()
    const str = await ldb.get('state')
    return JSON.parse(JSON.parse(str))
}

// 异步处理数据, 获得一个每条记录的数组
const getItems = async (onetabs) => {
    const items = []
    onetabs.forEach((ele, index) => {
        console.log(ele.id)
        ele.tabsMeta.map(ele => items.push(ele))
    })
    return items
}

// 异步连接mongo
const getMongoDB = async () => {
    const db = await MongoClient.connect(url)
    return await db
}

// 异步写入tab组
const insertGroups = async (db, onetabs) => {
    const dbo = await db.db("onetab")
    await dbo.collection("onetab_groups").insertMany(onetabs)
}

// 异步写入每条onetab记录
const insertItems = async (db, items) => {
    const dbo = db.db("onetab")
    await dbo.collection("onetab_items").insertMany(items)
}

// 生成用于备份的json数据文件和方便查看的CSV文件
const genExtFiles = async (items, onetabs) => {
    /*
     * 生成表头，\ufeff 是防止乱码
     * csv中以 `,` 换列，`\n`换行
     */
    let title = Object.keys(items[1])
    let csvContent = '\ufeff' + title.join(',') + '\n'

    // 添加表体
    items.forEach((item, index) => {
        let c = Object.values(item).join(',') + '\n'
        csvContent += c
    })
    // 生成文件夹
    await mkdir('onetab')

    // 生成csv文件
    await writeFile('./onetab/tab_items.csv', csvContent)

    // 生成JSON
    await writeFile('./onetab/tab_ori.json', JSON.stringify(onetabs))
    console.log('数据文件生成完毕, 请打开当前目录下的onetab文件夹查看.')
}

// 主函数
const main = async () => {
    const json = await getOnetabData()
    const db = await getMongoDB()
    const items = await getItems(json.tabGroups)
    await insertItems(db, items)
    await insertGroups(db, json.tabGroups)
    await genExtFiles(items, json)

    await db.close()
    console.log("写入MongoDB完成, 程序执行完毕. ")
}

main()

Result#

Finally, I wrapped a binary desktop application based on the recently popular Tauri.

Don't know Tauri yet?
See here: Throw Away Electron, Embrace Tauri Built on Rust

It Implements the Following Features#

• View all exported data in Onetab's original style
• Added pagination to handle large amounts of data
• Added date and keyword filtering

Coming Soon#

• Automated backup implemented in Rust, with no need to install Node separately
• Cloud sync via WebDAV or others
• Data analysis, word clouds, NLP

Repository#