Backups

Over the last few weeks I’ve been working on upgrading my home desktop computer, and I’ve documented much of the process here. New RAM, new CPU and cooling fan, several missteps along the way, but I got it all working nicely. Which of course meant that something new had to go wrong.

First, let me tell you about my current backup system for this box. For a number of years, every time I switched computers or OSes, I would back up the entire disk (or at least my home/user directory) to an external hard disk. And then I’d copy back all the stuff I needed, but kept the old copy around. Eventually, all of these copies got consolidated into my desktop computer. There’s all kinds of good stuff in there. Old documents, old code, old programs and downloads, photos, plans, projects, on and on.

In addition to that, I have a well-curated collection of audio, video, and ebooks. All told, this is about 800 GB of data. It all currently lives in my Dropbox folder, so it’s backed up to the cloud that way. And in addition to that, I do an rsync backup of my Dropbox folder to another 2TB drive.

The rsync job runs via crontab every night at midnight, and I’ve been using healthchecks.io to monitor that job. It’s a great service. You just add a ping to healthchecks when your backup (or whatever) job is done. If healthchecks gets the ping, all is good. If it doesn’t get a ping within a specified period, it alerts you. I have mine set up to alert me via email, but there are a ton of other integrations. It’s free to use if you just have a few personal jobs like this.

Generally, when I’ve gotten a ping on this, it turns out to be the Internet was down, or I dual booted into Windows and forgot to switch back before midnight, or the computer was just turned off (because, say, I broke my cooling fan mount). But the other week I started getting pings and my logs showed that rsync was failing to copy files to the backup drive. A couple of times I just ran the job again and it worked, but some files still had problems. Then, the other day I booted back into Manjaro after doing playing some Halo on Windows, and… well, actually I did NOT boot back into Manjaro. Because fstab was failing to mount my backup disk. I got into recovery and edited fstab and was able to boot fine. Mounted the disk manually and found 1,900 something errors. Luckily, it was just the local backup disk, so no data loss. Wiped it and repaired it, and it seemed to work fine. The errors are still there, but marked as areas of the disk to not use. But … yeah. This is the 2TB disk that I bought back in 2011 when I built this machine. It’s had a good life. But I’m not going to trust it for anything vital at this point.

I was going to buy a 2TB SSD for around $200. But decided on a new HDD for about a quarter of that price. The way I have my Linux system set up is with a 256 GB SSD on root /. Then my /home directory mounted on a separate 2TB hard disk. So I copied my whole home directory over to the new disk, wiped the old one and now that’s the backup disk. A bit of trial and error editing fstab, but once I got it right, it all booted up like nothing ever happened.

But it got me thinking about my backup strategy. Dropbox has served me pretty well, but it’s not a really professional backup system. They do keep copies of deleted files for a while, but you have no control of that. If I corrupt or change some file incorrectly, once it gets backed up, that’s that.

[Edit: Just discovered that Dropbox Pro, which is what I’m on, also gives you 30 days of file versioning, which is not too shabby. Other plans offer up to 180 days of versioning.]

My rsync strategy wasn’t much better. It was just overwriting and deleting copies to make a mirror of my source data.

I will note that any non-trivial code I write gets checked into github or gitlab or bitbucket on any major changes.

So I started looking into better tools. My first attempt was using restic and Backblaze B2. This will allow me to do incremental snapshots, so I can go back and get a previous good version of a file that might have been deleted or corrupted. Backblaze B2’s storage is pretty cheap. But they also charge for downloads. And I discovered that when you run a restic check or restic prune it makes a bunch of api calls to get file data and Backblaze counts these as downloads. This seems to have the potential of inflating the cost. But I’m not really sure how serious of an issue it will be until I try it and see the real world usage numbers and charges. It may wind up just being a few cents per month.

The other service I looked at is Wasabi. This service caters more towards enterprise accounts, but there’s nothing to stop you from using it personally. On the plus side, they only charge for storage, not downloads or any kind of data resulting from api calls. On the minus side, you have to pay for a minimum of 1TB of storage, and they have a 90-day retention policy, which means that if you upload a large file and delete it the next day, you still pay for 90 days worth of storage for that file. Not a big deal for a backup solution that is going to be a pretty stable size. And it’s only $5.99 for that 1TB, with $0.0059 per GB beyond that. Since I’ll be backing up close to 800GB, this is not bad.

So, I’m still figuring out what I want to do about all of this. For now, I’m at least switching over to restic for my local backup to give me some more security there. I need to clean up all my old archived files because there is a ton of duplicate stuff there. Then I’ll probably go with Backblaze for a month or two and see how that pans out.

XFCE

This is just a fun little thing I have been playing with the last couple of days.

As you may have gathered, I run Linux on my personal machined and after spending plenty of time on many different desktop environments, I keep coming back to XFCE as my favorite. It hardly changes year after year, so there are plenty of XFCE memes making fun of its nonexistent “exciting new features”, but it’s stable as hell, quite customizable and very minimal, which I like.

I use multiple workspaces (virtual desktops) and I use the workspace switcher panel plugin to indicate which workspace I’m in. By default, it looks like this:

I run four workspaces. The switcher shows a preview of what’s on each screen. The config is minimal:

I turn off the miniature view, so it just shows the number of the workspace, like so:

That’s fine, but rather boring. But it turns out that just about everything in XFCE is visually configurable via CSS. You just edit (or create) the file: ~/.config/gtk-3.0/gtk.css. Then you need to reload the thing that you’ve changed. The workspace switcher is part of the XFCE panel, so you can reload it with xfce4-panel -r. The tough part is knowing what the element names, ids and classes are. To help you inspect the “DOM”, there’s a GTK debugger tool. First, you quit the panel: xfce4-panel -q, then you restart it in debug mode by running GTK_DEBUG=interactive xfce4-panel.

This restarts the panel and opens up a bunch of windows – one for each gtk element running on the system. Here’s the one where my workspace switcher is:

You can see the XfcePanelWindow, the PanelItemBar, and under that the workspace switcher is the XfcePagerPlugin with XfcePagerButtons. There’s all kinds of views into what’s going on. This one I found most useful:

Now you can see each element, its ID and class, if any, and all the CSS properties for that element. Armed with this tool, some helpful tips from some posts on reddit, and a lot of hacking and slashing, I wound up with this in my gtk.css file:

#xfce-panel-toggle-button:checked {
  background-color: #cccccc;
  color: black;
}

#xfce-panel-toggle-button {
  font-size: 12px;
  padding: 0 2px;
  margin: 6px;
  background-color: #444444;
  border-radius: 12px;
  box-shadow: 0 0;
}

And when I reload my panel now, my workspace switcher looks like this:

This is life changing… not. But I like it and it was fun to do. Maybe I’ll mess with some other elements, but right now I’m pretty happy with how everything else already looks.

Bit-Box v2

No real significant updates here, but a few ideas I’ve been kicking around.

For one, I learned about how you light up keyboard keys. I assumed you had to buy switches that had LEDs in them. Maybe you can, but there are two other methods that are more prevalent. One is to just have a surface mounted LED under a clear switch or a switch that has an opening to let the light shine through. The other method is to insert the right sized LED into your switch. Little did I know that the switches I bought do actually have a little LED-sized cutout in the top, with two holes that go right through the switch to the back.

Some research led me to buying 100 3mm flangeless 3 volt LEDs for something like $5-6. They pop right into the switches like this:

Since they are 3 volt, they can run right off the Arduino micro board.

I think I have this mounted upside down, but whatever. I also think it might be too bright, so if I ever get to the point of adding these to all my switches, I’ll have to see how bright it looks and might need to add a resistor, or maybe even a knob to control the brightness. Or… I just thought, another control to have the Arduino to it via PWM. So many things to try.

I’ve also been thinking about alternate build ideas to get around printing switch plates each time. I printed out the switch plate template I have on paper, used it to mark up a piece of 1/4″ oak, and chiseled out a cherry-mx-sized square hole.

This worked surprisingly well and was surprisingly non-difficult. It fits snugly and securely. This opens up all kinds of other build ideas, which are mostly just floating around in my mind for now until I get back down to the workshop and actually try something.

What have I been up to over the last week? A few things.

RAM

I got my next 16GB batch of RAM in for my desktop. Out with the old, in with the new. It works. With the upgraded CPU and double the RAM, this 9 year old computer feels like new.

Now I have an extra 32 GB of DDR3 RAM that I will probably never use.

The four on the left didn’t work in my computer, but I suspect it might work in some other motherboard. The rest of it all works fine. It’s going to end up in a drawer somewhere, so I’d love to get it to someone who can use it. Open to any offers. Shipping, plus whatever you want to pay. I’m not looking to profit, but if I could recoup some fraction of what I spent, that’d be nice.

Bit-Box

I was really enjoying my Bit-Box for the last week or so. Until I was messing around with the USB cable for some reason and managed to tear the connector right off.

The problem, I think, was that I used way-too-heavy, solid wire to connect everything. Everything was far too stiff so when I put a bit too much strain on the USB cable, the connector was the first thing to go. Also, soldering everything directly to the board wasn’t the greatest idea either. So the first step was to remove the old one…

Then I soldered on the pins that came with the Arduino boards. I had some ribbon cable with connectors on it, so I cut some of that to size and plugged it onto the pins.

Now it’s much more flexible and in the event that something does go wrong with this board, I can easily pull it and put in a new one.

I still have plans on making a larger one, but haven’t started anything on that front yet.

Attractors

I’ve been continuing to work on learning GTK3 desktop Linux apps, working on making an app that does live drawing based on changeable UI parameters. It’s been fun and I’ve made some good progress. I figured out enough of the architecture to feel some confidence in how to set something like this and started in on making a real app using this idea.

One app that I’ve done multiple times over the years is what I call “Strange Explorer”. It presents different formulas for strange attractors, allows you to change their parameters in real time and see and save the outcome. I’ve done it in Flash and JavaScript, and even made a native iOS version. Here’s what I’ve come up with so far:

Two things about this project that I’m really happy about so far. One is that the different formulas are created as external configuration files that get read in at run time. They look like this:

name Hopalong Attractor
x_formula y-1-sqrt(abs(b*x-1-c))*sign(x-1)
y_formula a-x-1
z_formula n/a
scale 10
iter 50000
init_x 0
init_y 0
offset_x 0
offset_y 0
rotation 0
a -0.537
b 2.797
c 2.973
d 0

The code reads all the files in the formulas directory at startup. So to create a new attractor, I just have to create the new config file for it and start the app. It will automatically populate and function.

The other part is the formula parsing itself. Note that the formulas in the above configuration file are just lines of text that populate text fields in the UI. And those text fields are editable. And yet, it winds up as runnable code. To accomplish that, I’m using a library called tinyexpr. https://github.com/codeplea/tinyexpr You just feed it a string of text and it is evaluated. There are ways to bind variables to it so a, b, c, x and y are all plugged in to the formula at run time as well. It handles all the standard C math library stuff. But you can also define your own functions, like I had to for the sign operation in that example.

Eventually, I’ll create a way to bookmark settings and define brand new formulas right in the UI. Having a lot of fun with this project though.

Interpreters

To be honest, the attractors project has been a bit sidelined during the latter half of the week, as I’ve started down a whole new giant rat hole.

When I was creating the configuration files for that project, I needed a way to read and parse them at runtime. I thought about JSON and YML. There are C libraries that handle those. There’s also libraries that are totally designed for reading configuration files of various formats. In the end, I decided all of those were overkill for what I needed. I just needed to read lines from a file, separate the first word (name of property) from the rest of the line (property value) and add that to a model object.

So I wound up making my own parser. It does just that – reads each line, splits each line on white space. Takes the first element as the property name and joins the remaining elements as the value. If it’s a numeric property, it converts it to a number. Super simple, but it works fine. I’ll need to bolster it up a bit with some error checking, but it’s all I really need.

But it reignited my interest in building an interpreter. And reading up on tinyexpr fanned those flames a bit more.

I’ve had this idea for a while now to build my own DSL (Domain Specific Language) for creating graphics. It would sit on top of something like Cairo Graphics, parse a simplified source file and create a picture. Mainly so I could do away with a ton of setup and boilerplate and just write the commands that make the pretty shapes. It would probably wind up looking something like Processing, but even simpler.

Now before you start throwing links at me to projects that already exist like that, or ask why don’t I just use ____? … that’s not the point. The point is that I want to make something like this. Even if it’s only for myself.

A while back, when I was doing more with Go, I picked up two books:

Writing an Interpreter in Go

Writing a Compiler in Go

I got a little ways into the interpreter book, but never finished. Not a criticism on the book, which seemed pretty good actually, but I got distracted.

Anyway, I decided to go back into this idea and found a few really good online resources. The one I’m working through now is Let’s Build a Simple Interpreter.

When I first started this series, I assumed it was maybe up to five parts and I saw that it was started 5 years ago, so figured I’d get through it fairly quickly. It turns out that it’s now up to 19 parts and the last installment was earlier this year.

And it is amazing.

As an author of technical books myself, I really appreciate it when someone can explain complex subjects clearly. There are so many poorly written technical books and tutorials, when you find one that is really well done, it is a joy. This is one of those.

Writing an interpreter is a complex thing. He takes it one step at a time, in bite sized chunks that you really can wrap your head around. The overall goal is to write an interpreter that will handle most of the Pascal language. I’m on lesson 9 right now. And I don’t recall being this excited over some technology in a while.

This series focuses on Python as the language that the interpreter is being developed in. But I’ve been re-writing it in C. Porting it while learning it has been a great way to force myself to really fully understand what each line of code is actually doing. Much better than just copying and pasting. In fact, I think from now on, if I want to learn something that is not totally language dependent, I’m going to seek out tutorials in a language other than the one I’ll be using.

If you’ve ever thought about learning more about this subject, this particular series is a great place to start. Along the way I’ve found some other resources that I plan on looking into when I work my way though this one. I’ll list them here.

Sites:

• Let’s Build a Simple Interpreter
• Crafting Interpreters
• Let’s Build a Compiler
• Let’s Make a Teeny, Tiny Compiler

Books:

(These were all taken from the simple interpreter series I’m working through now)

• Language Implementation Patterns
• Writing Compilers and Interpreters
• Modern Compiler Implementation in Java
• Modern Compiler Design
• Compilers: Principles, Techniques and Tools

I’ve been working on a custom keypad build. I think I’ll call mine a Bit-Box. In my post last week I shared the initial proof of concept breadboard build and the second prototype build. This past week I finished the “final” version. I put “final” in quotes because I’m already thinking of how to make a better one. Anyway, here’s how it came out:

• 

• 

The previous iteration used an Arduino Micro clone, a plastic project box and some push-button switches. For this one, I wanted a nicer enclosure and better switches. I got some Cherry MX keyboard switches from Amazon. A dozen for around $10. And ten X-Keys customizable key caps for about the same amount. These are clear key caps with a top cover that snaps on. You can pop the top off and put a slip of paper in there to create a custom key.

I found some templates here: https://xkeys.com/xkeys/accessories/customprintedlegends.html Just go down to the section entitled “Templates for Pre-cut Legends”. Pull one of those templates into your favorite image creation program and create the keys however you want them to look and print out at 100%. Cut them out and they should fit just right.

Next, I needed some way to mount these switches and keep them in place. There are two methods generally. One is to get or create a custom circuit board with holes drilled in it. The bottom of each switch has plastic nubs as well as the leads sticking down. Those go in the holes and you solder each switch in place. The other method is to have a plate with square holes in it. The entire key fits through the hole and snaps in place. I couldn’t find anywhere to buy such a switch plate, but I did find a model for one at Thingiverse: https://www.thingiverse.com/thing:2789684/files

I downloaded the model and had it printed at Shapeways. I printed it on the cheapest plastic and it was very reasonable. They kill you on the shipping though. Would be better to buy in bulk probably. Here’s the plate I got, with the switches inserted:

They snapped right in perfectly. Then, I put the key caps on…

And then printed up the inserts and inserted them.

Top row is to switch to workspace 1, 2 and 3, close the current app, close current window. Bottom row is terminal, firefox, slack, clementine (music player) and file manager. I was pretty excited about how this was looking so far. Now on to wiring it up.

I got a few more Arduino Micro clones and soldered wires to one of them and connected those leads to the switches. One switch leg went in common to the ground terminal of the Arduino. And one to each of 10 input terminals.

• 

• 

At this point I was able to plug it in, send my program to it and test it out. It works! Now I wanted to create a nicer enclosure.

I got some 1/2″ oak and set about fabrication. I’m a bit rusty on the woodworking skills, but it all started coming back to me. Ruined a few pieces by measuring wrong, but got there in the end. This was a bit more challenging than a lot of my other woodworking projects because it the inside of the box has to be exactly 2 3/4 x 1 1/2 inches, with fairly tight tolerances, in order to fit the switch plate. That took some careful math to but the outside dimensions of each piece so that the inside would come out right. Not rocket science, but easy to mess up, as I proved.

• 

• 

• 

• 

• 

• 

I cut some grooves in the side pieces and inserted some rails in them. The switch plate goes down into the top of the box and sits on these rails. It worked perfectly. I left the bottom open and filed a groove on the bottom of one of the sides for the cable to pass through. All very low tech, but I like that there are no screws or other fasteners holding the guts in place, and nothing to disassemble to get the switches out if I need to. Just turn it over and they come right out. I didn’t put any finish on the wood yet. I might do some shellac later, but I like the look of raw oak.

• 

• 

Post Mortem

What went well:

• Key caps, switches, switch plate, Arduino, box build.

In short, I’m really happy with how it turned out.

What could have been better:

• I used solid wire, probably too heavy for the purpose. Made it hard to arrange everything. If I do another one of these, I’ll use lighter weight stranded wire. Maybe think about using some connectors.
• The box came out a bit chunkier than I want. It could easily have been half the height. (I am thinking about cutting this one down in fact.) And I’d probably go with 1/4 or 3/8 inch rather than 1/2 inch wood next time.
• The graphics look great, but maybe a bit washed out. I might try printing them on photo paper to have them be a bit more vibrant.

Thoughts for next time:

• I want to make a larger model, with maybe 15-20 keys. This will mean designing a custom switch pate, which might need more support in the middle. It also means altering the circuitry and software to use a matrix rather than have each key tied to its own pin. That will be a fun learning process.
• I might also look into white LED switches and print the graphics with a white background so the light shines through.
• It might be fun to think about some other controls. A rotary control for volume or brightness or whatever. Toggle switches for… something? Just go all out steam punk on it.

A lot of things going on this week, so I’m probably going to have to make more than one post.

This one will be about some general computer upgrades I’ve been making.

Dell XPS 13 Battery Replacement

So, I bought a Dell XPS 13 back in 2016. It’s a sweet little computer. Really thin, lightweight, almost no bezel. But in 2018 I was really yearning for another Thinkpad and got myself a T-480, which I really love. I haven’t been using the XPS much but felt bad about that since it really is a great machine. So I pulled it out and discovered the battery was totally dead. It would run fine if it was plugged in, but the battery would not charge at all.

My wife had a Dell laptop some years ago and exactly the same thing happened to it. And I’ve heard similar stories from others. Seems to be a thing with Dell.

Well, I went on Amazon and found a replacement battery. Checked out some videos on line that made the replacement look easy. Turns out, it was really easy. Nice job on that Dell. Half a dozen screws to take the bottom panel off. Then 4-5 screws to remove the battery. I took some photos so I’d know that I was putting it back together right.

It all worked just fine and now it’s fully operational. If I ever get back to going into the office, this will be the perfect device for creative commuting.

Desktop Upgrade

OK, that was the one that went well. Next project was my desktop. I built it from scratch in 2011. It had an i3 540, 4 GB RAM and a 2TB spinning rust disk.

Here’s the post on my old blog from when I first put this machine together:

Ubuntu and Me, Happy Together.

Over the years, it’s seen a lot of upgrades. I boosted the memory to 8 GB. I think the power supply died at one point. I know there’s a newer one in there now. And recently I added an Nvidia GeForce 1050 graphics card. All kinds of storage in there now. At this point it boots Manjaro Linux XFCE from a 256 GB SSD, with a 1TB drive mounted as the home directory, and another 2TB drive as a backup. Then another drive that dual boots into Windows 10. Despite its aging specs, it’s been totally functional and I don’t really have any complaints about it for what it is. Mostly a home server, file storage and backup (also have offsite backup), sometimes media server, general browsing and email, etc.

But things can always be better and I’ve been thinking for a while about upgrading it completely. New motherboard, CPU, RAM. That was going to run $300-400 for what I wanted. But I realized I could go from the i3 540 to an i7 870 and upgrade the memory to its max of 16 GB, for just around $120 total. So I bought those items and they arrived this week.

The memory arrived first and that was easy to pop in. The computer booted right up and I’m pretty sure it was notably faster than before. Until it rebooted itself about a minute in. Repeatedly. Ran a memory test and that crashed it consistently. Tried swapping out different sticks to see if maybe it was just one bad one, but no luck. From what I can tell, it’s either bad RAM or just incompatible. I’ve read something about low density vs. high density RAM and apparently my old motherboard likes the low density stuff. So out with the new and back in with the old, at least for the time being, because my CPU was now here.

I popped off the fan, took out the old CPU, cleaned things up, put in the new CPU, got some thermal paste on there, and struggled to get the fan back in. It’s the fan I bought 9 years ago. It’s cheap and has those expanding plastic tabs that go through holes in the board and snap in. Well, in my case, they snap OFF. Two of them broke. I was hoping I still had good enough contact with the heat sink.

• 

• 

It booted up OK, but it got real hot real quick and shut itself right down. So, I spent a day without a desktop, while I waited for a new fan to arrive. I was just hoping I hadn’t fried the CPU.

The new fan came the next day (thanks Amazon!) and has a back plate that goes behind the motherboard and the heat sink/fan screws into that. This meant I had to take the whole computer apart and get the motherboard out. More than I was hoping to have to do, but OK.

I gave it a good cleaning and did some better cable management while I was at it. Put it all back together and booted it up. I mixed up the SATA cables a bit, so had to go into BIOS and let it know which drive I wanted to boot from. Once that was done, it booted right up. I also mixed up one of the audio connections, but it was easy to fix. All running good now, and nice and cool! Can’t say definitively that it feels a lot faster. But I feel good knowing that it’s theoretically faster. 🙂

Now that I knew I hadn’t totally melted the CPU, I did a bit of research on RAM and ordered another 16GB which is specifically compatible with my system. That will arrive next week.

But later the same day I was rummaging through a box of old cables and gadgets and found two 4 GB sticks of RAM. I vaguely recall getting them some years ago and remember that they were causing the computer to crash, which is why they wound up in a box. But I figured I’d try them out again. So far, so good. I’ve been running with 12 GB (4, 4, 2, 2) for a few days now and it’s been rock solid. Did some stress testing seeing how much RAM I could use at once, and it was all just fine. I’m not sure what was wrong before. Maybe I had put them in the wrong slots or something.

Next Tuesday, I have another 16 GB coming in. I’ll own 48 GB of RAM for this computer – 3 times as much it can hold. And it’s all old DDR3, which is kind of useless in any new system I might build. Oh well.

So, a rough start, but pretty happy with where things wound up so far – even though with the second batch of RAM and the new fan, I’m closer to $200 in on this round of upgrades.

Of course now after taking out the motherboard and putting it all back together again, I’m going back to the idea of getting a modern board and CPU. And more RAM (weep). And possibly a new case. Maybe a project for later in the year.

I’m looking at the B550 boards that have come out recently. I used to build my own computers and always went for AMD. But I’ve been all Intel for the last decade.