All posts by Dave

Accessing the Zyxel C3000Z web admin interface after enabling transparent bridge mode

The Zyxel C3000Z DSL modem/router (at least one I lease from CenturyLink) is able to be placed in transparent bridging mode by going to Advanced => WAN settings => ISP Protocol and selecting Transparent Bridging. This turns off the router and wireless functionality and makes it function like “just a modem”. After doing this, I can use my own OpenWRT-based router/access point to connect with the WAN port set to PPPoE. It’s been quite a stable configuration for a while now, but I just experienced an afternoon of connection problems leading me to want to inspect the modem’s system log.

After enabling transparent bridging on the DSL modem, it will no longer run a DHCP server, meaning that plugging into it via Ethernet will no longer give you instant connectivity to its web-based administration interface. However, it will keep its old IP address and be accessible if you manually set your IP address to be in the same subnet. In my case the modem came set to 192.168.0.1, so I gave myself the address 192.168.0.2 and browsing to http://192.168.0.1 got me to the admin interface. If you don’t know the IP address of the DSL modem, it may be printed on a sticker somewhere on its case.

You can do a similar thing without having to physically plug in a new cable by adding a virtual interface to your own router using the same physical port that you use for the WAN connection. Assign this virtual interface an IP address in the same subnet and you should be able to browse to the C3000Z modem admin page. Just be sure to firewall it correctly.

Remote desktop on Raspberry Pi with Vino in 3 steps

If you run a GTK-based desktop environment on your Raspberry Pi, getting remote desktop working with Vino is extremely easy.

Step 1: install the package:
sudo apt-get install vino

Step 2: Configure the server to not prompt to allow connections and use an empty password:
gsettings set org.gnome.Vino prompt-enabled false
gsettings set org.gnome.Vino vnc-password ''

Step 3: Start the server:
systemctl --user start vino-server

Now you can connect using any VNC capable client.

BoossJuice hot sauce

For the past year or so I’ve been making my own brand of homemade hot sauce called Booss Juice. It’s based on pineapple juice, unfiltered apple cider vinegar (sorry Mikey! I like the tang), rehydrated arbol chili peppers, garlic, onion, and whatever selection of fresh hot peppers I have on hand. I’ve also sometimes thickened it with guar gum. My favorite batch had Pequin peppers that I grew from seed into bushes. These days I usually throw habanero, jalapeno, and most recently cherry bomb peppers into the mix. I blend it, filter it through a sieve, and bottle it without any kind of pasteurization or cooking. This leaves the heat from the raw peppers more intact, but leads to a shorter shelf life.

Debian on the Lenovo ThinkPad A485

Here are some notes about getting Debian up and running on the ThinkPad A485.

Install these missing firmware packages:

apt-get install firmware-amd-graphics firmware-realtek

..or download them onto removable media from another computer:
https://packages.debian.org/search?keywords=firmware-amd-graphics
https://packages.debian.org/search?keywords=firmware-realtek

With kernel 5.x the r8822be Wi-Fi driver kernel module is replaced by rtw_pci / rtw88 which needs missing firmware (Debian bug report):

wget -P /usr/lib/firmware/rtw88 https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/rtw88/rtw8822b_fw.bin
modprobe -r rtw_pci
modprobe -r rtw88
modprobe rtw_pci

Bluetooth will choke when using Laptop Mode Tools due to runtime PM. Blacklist the ID of the USB Bluetooth controller by adding this line to /etc/laptop-mode/conf.d/runtime-pm.conf:

AUTOSUSPEND_RUNTIME_DEVID_BLACKLIST="0bda:b023"

Blacklisting client mode BSSIDs in OpenWRT

I have an OpenWRT router that I use to privately share a connection to a public hotspot. In other words, I use one of the Wi-Fi interfaces to connect to the internet via a hotspot, and share it to my own devices via another Wi-Fi interface. This is pretty easy to accomplish with OpenWRT and simply requires switching the Wi-Fi interface to client mode and setting the network interface to something in the WAN firewall zone (typically wwan).

This post isn’t going to cover the basics of that configuration, but instead cover a problem I’ve had with hotspots that associate but then fail to respond to DHCP requests. That essentially kills my internet connection for a while until udhcpc times out (sometimes it doesn’t) and a new association attempt is made with a different access point. Each hotspot has its own BSSID, which is the MAC address of the access point. You can find this address either from the Wireless section of the web frontend, or from watching the association attempts from the system log (e.g. logread -f). There is no obvious way to blacklist a particular access point by MAC address in client mode from Luci (OpenWRT’s web frontend). It is pretty easy to accomplish from the command line. First, determine which radio you are using for the client mode connection, e.g. radio0, radio1, etc. and then find its configuration “section” name, e.g. “wifinet1”, “wifinet2”, etc.

wifi status

You can also get this information another way using the uci command:

uci show wireless

Once you have found the configuration section for the radio device you are using, it’s as simple as adding the blacklist value. Here we will assume the configuration section ‘wifinet1’ and set the blacklist to include the fake MAC address values ‘xx:xx:xx:xx:xx:xx’ and ‘nn:nn:nn:nn:nn:nn’:

uci set wireless.wifinet1.bssid_blacklist='xx:xx:xx:xx:xx:xx nn:nn:nn:nn:nn:nn'

As you can tell, the MAC addresses are separated by spaces.

Now write the changes to the system configuration:

uci commit

Finally, restart the wireless interface. We’ll assume radio0 as an example:

wifi up radio0

This will prevent the wireless client interface from connecting to the access points in the blacklist. This works because the bssid_blacklist value is passed through from uci to wpa_supplicant via hostapd.sh. Note that this value may get clobbered if you touch the wireless interface configuration from the Luci frontend since it is not included there.

btrfs inode errors with file extent discount / file extent holes

This was a new one for me. An external USB 3.0 btrfs RAID 0 volume had some random I/O errors of undetermined cause during heavy writes — no kernel oops or obvious bug, and no clear hardware failure either. It came after seeing a bunch of these in my kernel log:

xhci_hcd 0000:00:14.0: ERROR unknown event type 37

This apparently has something to do with filling a ring buffer according to this LKML post. I’m not sure if it’s related but it was coincident with the following errors.

I’ve been running btrfs on a number of volumes for years (with many previous catastrophes) and this is the first time I’ve seen this particular error:

btrfs check --repair /mountpoint
...
root 5 inode 7553112 errors 100, file extent discount
Found file extent holes:
start: 0, len: 16384
root 5 inode 7553113 errors 100, file extent discount
Found file extent holes:
start: 0, len: 12288
root 5 inode 7553115 errors 100, file extent discount
Found file extent holes:
start: 0, len: 28672
...
root 5 inode 7555752 errors 100, file extent discount
Found file extent holes:
start: 0, len: 45056
ERROR: errors found in fs roots
found 9165863124992 bytes used, error(s) found
total csum bytes: 8933283920
total tree bytes: 15941156864
total fs tree bytes: 5797117952
total extent tree bytes: 473710592
btree space waste bytes: 1645027279
file data blocks allocated: 9149921968128
referenced 9216958279680

The solution I came across is a bit time consuming. It relies on freeing the affected inodes, which in turn means removing the original files from the filesystem. You can copy them elsewhere or just delete them if you have a backup or decide they aren’t valuable. In my case I kept the files around long enough to see if they matched a backup — and they did, indicating that the files themselves weren’t affected by this problem.

First I took the output from btrfs check and picked out the inode numbers:

grep 'file extent discount' btrfs_check_output | cut -f4 -d' ' > bad_inodes

Explanation: I put the output from btrfs check into a file called btrfs_check_output. The grep command searches this file for the lines with inode numbers, and cut picks out just the inode numbers from the rest of the line: -f4 means take the fourth field -d' ' delimited by spaces. The output goes into a file called bad_inodes.

This ended up with a list of 705 bad inodes! Sheesh… now comes the long part. In order to find the files associated with these inodes, I put this Bash command line together:

while read inode; do find /mountpoint -xdev -inum $inode -print0 -quit; done < bad_inodes > bad_inode_files

Explanation: This reads the bad_inodes file as standard input and loops over each line as variable $inode and then executes a find. The output is sent to a file named bad_inode_files. The arguments to find are -inum which searches for a file by inode number, and -xdev which stops the file search from descending into other filesystems, since we don’t want to mistakenly find files with the same inode number on a different filesystem. The quit option stops the search after the first result, since we don’t need to keep searching the entire filesystem after we have located the file associated with a particular inode. -print0 is optional and is for handling filenames containing whitespace by delimiting each item of output with a null character instead of the default newline. You can substitute -print0 with plain old print if you aren’t going to process this output with another utility, or if you are sure there aren’t filenames with whitespace in your output.

This took a while to complete since it has to search for each inode starting at the root of the filesystem, and in this case it’s a 10TB array and 705 inodes. When it completed, it produced a list of affected files in a text file named bad_inode_files.

I could have just moved or deleted the files, but I wanted to inspect them first. The affected filesystem is a backup, so I compared the files to the originals. All of the files matched according to rsync which means that they’re probably not damaged. I then went ahead and deleted all of the affected files since I had them on a backup. If I didn’t have a backup I could have moved them to another location and then back.

Now after removing the files with the affected inodes, btrfs check returns no errors.

OverDrive Account No Longer Able to Be Used for Library Sign In

OverDrive for Libraries recently changed their platform to no longer allow sign-in with an OverDrive account. The end result is that you must log in with your library card number and PIN code instead. This change took place with no public announcement that I could find on their website or social media channels. The “What’s the difference between an OverDrive account and a library card?” help article removed this language sometime between 13 AUGUST 2019 02:50 PM and 19 AUGUST 2019 02:06 PM:

An OverDrive account:

Lets you sign into your library’s digital collection

Can be used to sign into multiple libraries

Kombucha

This post will be my braindump about brewing the ‘booch! This is going to be a quick and dirty first draft that I’ll keep editing as I’m short on time at the moment.

Kombucha is not hard to make. Starting from scratch, get any old bottle of booch and drink most of it. Leave about a cup from the very bottom with the cloudy bits for starting your own SCOBY.

OH MY GOD WHY IS SCOBY IN ALL CAPS?!

It’s an acronym — Symbiotic Culture of Bacteria and Yeast.

This is the thingy that “brews” your booch and looks like a weird jelly blob floating (hopefully floating — sometimes it can sink) at the top of your jar. It can get weird colored snot, usually brown, coming off of the bottom of it and this just indicates that it is happily munching on sweet tea. If you see weird colors that aren’t caused by air pockets or bubbles on the top, abort! Start over from scratch or with a different SCOBY. You don’t want to drink that.

In order to brew kombucha you need a SCOBY. But you can make your own SCOBY from almost any bottle of kombucha as long as it’s really kombucha and not some kind of trendy kombucha-like drink or something heavily filtered. It should have sediment at the bottom. Add the bottled kombucha to this recipe and you’ll have your own SCOBY after a couple of weeks. The first brew will be crappy but possibly drinkable. After a couple of brews your SCOBY will be thick and happy, and ready to share with friends for their own brews.

Required stuff:

  • Plenty of loose leaf tea. I use a mix of both generic black tea and gunpowder green teas. If you are a tea snob, save your good tea for something else. Unless you want something really exotic like Pu-Erh flavored kombucha, in which case, go nuts!
  • Sugar. I use organic sugar because it’s pretty much guaranteed to be vegan (some refined sugars are processed with bone char).
  • A clean half gallon jar or two. One for the initial brewing of tea and one for straining off the loose leaf tea. Don’t use metal or ceramic for brewing kombucha since they will react and can leech into the brew. I would not recommend plastic either since the acidic nature of kombucha brewing may end up leeching chemicals too. Your call. If you have cleaned a jar by hand make sure it is extremely well rinsed. You don’t want any kind of cleaning product to end up in your booch. If you only have one jar you can brew the tea in anything large enough that you’re comfortable dumping boiling water into — so again, probably not plastic.
  • Some breathable material to cover the jar: a dish cloth, paper towel folded in half, coffee filter, thick cheesecloth, bandana, or other semi-permeable material that is ideally very fine so that it cannot be penetrated by insects, but importantly, is not airtight.
  • A metal ring, rubber band, or elastic of some sort to keep the covering material tight on the mouth of the jar
  • A sieve or strainer to strain out the loose leaf tea, and to strain off any solids used for flavorings at the end of a batch.
  • A funnel for bottling the finished product

The process:
-Boil about half a gallon of water
-Add one tablespoon each of loose leaf black and green teas
-Let the tea brew for a few minutes and then strain it into a half gallon jar
-If you poured in a full half gallon, pour out about a cup (or however much kombucha you have to add in)
-Stir in 1/2 cup sugar until fully dissolved
-Wait for the tea to cool down to room temperature. If you do not wait you will fry the kombucha that you are about to add in.
-Add in about a cup of kombucha (from your last batch preferably, or from another source)
-Make sure there is enough clearance to fit the SCOBY and a few centimeters of headroom for air at the top of the jar. If there is too much liquid just pour some off, or scoop it out with a measuring cup to prevent having a sticky jar.
-Add your SCOBY on top. It should float. If it sinks it may be because the tea is still too hot, or it’s a very thin SCOBY, or it’s feeling like, down, today. No big deal. It may come back up or you may end up with another baby SCOBY on top at the end of this brew. Either way it will do its job wherever it is.
-Move your jar to wherever it is going to stay for the duration of the brew. It should optimally be somewhere that is warm (but not hot) and dark, with minimal chance of attacks by bugs. It should NOT be placed near any other open fermentation projects like ginger beer, etc. to prevent cross pollination (is that the right term?).
-Once the jar is in place, add your covering material and secure it with an elastic or metal ring. Double check the seal. Make sure the covering material did not get wet while manipulating it because that can lead to mold. Double check that the SCOBY is not touching the covering material. That can dry out the SCOBY and/or lead to mold. Triple check the seal. You don’t want fruitflies finding their way into your brew.
-Wait about five days before tasting a sample of your brew. I tend to use a reusable straw to push past the edge of the SCOBY and take a quick taste. Only you will know when it tastes right. Don’t ask me. It should have some sourness to it and not be sweet. If it’s sweet let it keep going. If it takes over a week your SCOBY may be immature and just need more time for a brew, or your temperature may be low and that will slow the fermentation process. If the ambient temperature is really hot you may have a usable batch in less than 5 days, but watch out, because when you bottle it it may also carbonate very quickly as well and lead to exploding bottles.
-Once the taste is to your liking, remove the SCOBY with clean hands and put it on a clean plate for temporary storage. Dump about a cup of the brew into a sealed container for your next batch. If you’re not going to make a new batch right away, put the SCOBY in with it, seal the container up, and put it in the fridge. That’s called a “SCOBY hotel”. It’ll keep for a while — even longer than a while if you give it some more fresh (but cooled to room temperature) sweet tea to munch on while it’s resting.
-Add whatever flavorings you want at this point. My go-to flavoring is fresh ginger and turmeric that I grind up on a microplane and add in with no particular measurement involved. I then let it sit for around 24 hours to soak up the flavoring, but what you use may require more or less time to absorb the particular flavor you are going for. It will take experimentation.
-If you’ve added stuff for flavoring, you’ll need to strain off the kombucha before bottling it.
-I like to use existing kombucha bottles since they’re smaller portions and opening one bottle doesn’t release all of the nice carbonation for the whole batch
-I also like to use at least one plastic bottle per batch to judge the carbonation level by squeezing the bottle to see if it’s hard or not. Once the plastic bottle is rock hard, I know the batch is carbonated and ready to be chilled.

That’s it for now, I’m sure I’ve left out plenty. I hope it’s helped a bit, and I’ll circle back and keep editing this as time permits.

Caring for your yeast baby

In order to keep a sourdough starter (“yeast baby”) viable for baking purposes, it is necessary to feed it regularly. The more consistent you are with the feeding schedule, the more reliable your starter will be for baking delicious breads (and other things, too).

Required stuff:
-Jar of sourdough starter
-Clean spare jar
-Kitchen scale that weighs grams
-Bag of unbleached white flour
-Container of filtered water

The golden rule is a 1:1:1 ratio — to take one part by weight of the existing sourdough starter and add to it one part flour and one part water. Mix that all together into a clean jar until there are no dry lumps of flour. I like to use a regular table knife to mix with since it’s easy to scrape off.

If you are not going to be baking in the near future, use a minimal weight, like 15g of each the existing starter, flour, and water. Keep that jar in the refrigerator for up to a week before you feed it again. You can use the leftover starter to make pancakes, start a backup batch, or just throw it out.

If you are going to be baking, you will need to figure out how much starter the recipe calls for and divide that by 3, with some left over for continuing the starter. For example if we wanted to make my standard sourdough loaf, it calls for 120g of starter, which requires 40g starter from the jar, 40g flour, and 40g of water (40+40+40=120). But we’d also want about 15g extra starter to keep the starter going in the jar. Weigh out whatever you have left from the jar of starter, whether it is 15g or less, and add equal parts flour and water to it. Be aware that if you end up with less than 5g of your starter to continue on with, you may fundamentally change the character of it going forward.

After feeding, let the yeast baby sit for two to five hours until it has doubled. It is called “fed” starter after it has roughly doubled. This is when you should put it to use.

Things can go wrong. I tend to keep at least one backup jar of starter in the fridge in case I break the jar, or somebody mistakenly throws it out, or I feed it almond flour because I’m too tired to realize what container is in my hand.

If your sourdough starter starts developing odd colored blotches, it could be mold. If this is your only starter and it is mature (usable for baking), take a portion of an unaffected area and start again with that. It may have been caused by contamination or too warm an environment. It could also be weird blotches from using a flour that isn’t entirely pure, like a malted flour or enriched flour. If that’s the case, just switch to using just 100% pure unbleached white flour.

If your yeast baby develops a layer of liquid along the top, it is alcohol and you are starving it of precious nutrients. Pour off the alcohol and give it more consistent feedings. It should recover, but it may taste different.

Temperature will impact the kind of yeast that dominates. You may find that you prefer the flavor or behavior of a sourdough starter that’s kept mostly in the fridge versus one kept mostly at room temperature, or vice versa. You also can slightly vary the amount of water away from a 1:1:1 ratio to favor different types of yeast. Knowing the specific impacts on types of yeast is beyond my expertise, so I would encourage you to turn to your own research if you’d like to know more about that.

Starting a sourdough culture

Starting a “sourdough starter,” “sourdough culture,” “mother,” or “yeast baby” as I prefer, just takes some patience and persistence. You’ll need a kitchen scale to weight grams and two clean jars. One jar will hold the current batch of starter, and you will transfer just a bit of that starter into a new batch with some equal parts water and flour in order to “feed” it. Here’s a summary of what I did:

From scratch, find a clean wide-mouth jar, and rinse it thoroughly so that there is no detergent left in it. Mix in 15g flour and 15g water until it is roughly consistent in texture with no dry clumps of flour. Put it somewhere warm and dark. Wait a day. Transfer 15g of this mixture into a clean jar and add 15g flour and 15g water to it. This is called “feeding.” Keep repeating this cycle until you think you’ve lost patience. At first it will smell like boring dough, and then maybe after a week or two you’ll hit this really, really funky smell, like rotting wet socks at the bottom of a garbage can. But keep going through this stinky phase — it will start to mellow out and develop into something that still smells sour but not offensive. When you think it may be starting to mature, probably around week 3, draw a line at the top of the mixture and check back a few hours after you’ve fed it and see if it has doubled in height. If it has doubled, you’re done starting the starter! If not, keep at it, because it may take a bit longer to develop into something usable for baking.

Some people claim that there are shortcuts, like using different kinds of flour mixed in (particularly rye), or a bit of fruit juice of some sort (like pineapple). I haven’t tried these because I just hammered on with the above pattern for a month and got a really nice yeast baby that’s been with me for a few years now. My next post will be about continuing care of your yeast baby.