Posts in Article
Add Timestamp to Videos
I am involved with some some experimental work and a lot of the video footage is recorded with a GoPro. They are durable and work well in the field. Unfortunately they don’t support timestamp overlay with the date and time on the video. I decided to use FFmpeg and a shell script to automate the process. The following script will take the video, extract the
creation_time tag from the video and use that to generate the timestamp overlay.
Python - Loops and Exception Block Else Statements
The try/except block has an option else clause. That
else clause is executed if an exception is not raised in the block. Loops, also have an else clause. I never thought that I would actually need to use those and thought they were superfluous. Today, I used both. In the following code, I wanted to create a folder, but wanted to make sure that I didn’t create a duplicate folder (i.e. I didn’t want to write files into the same folder).
Windows Terminal and Cmder
Using windows to develop can be a bit of a challenge. It doesn’t have any good tools for cross-platform python tools. On Linux, I use make and a
makefile to orchestrate building and configuring virtual environments. Clone the repo and
make venv and I have a functional and repeatable environment. A few months ago I discovered Cmder. I learned that it has
make and most of the tools I use out of the box for windows. The only issue, it is a bit of a pain. Recently I decided to try installing windows terminal and host Cmder in that. That works really well and seems to be pretty stable and is relatively easy to install. It would be nicer if it was an automated install, but these instructions are not too bad.
Uncertainty Propagation in Calculations
In science and engineering, uncertainties and errors are a fact of life. This post is a study of how uncertainties can be used in calculations. More importantly, this post explores how uncertainty is propagated to derived variables.
I have written a simple fuel tracker application and you can find it here. The idea is a simple system to keep track of fuel records far various vehicles I have owned through the years. I have been keeping track of my fuel records since 2002 across 4 vehicles. I have over 800 records stored in the database. A modest amount, but good information.
LED Strip Calculations
The idea is to construct a spiral (Archimedean spiral or others) around a right-cone simulating a Christmas tree. We want to model the situation and understand how many lights or how long the strip(s) should be to wrap the proper amount of loops around the tree. This blog will establish the basic model and mathematics. This article will walk you through the mathematical derivation and the calculations. The derivations are for completeness. An understanding of the process is not required to use the results.
Configure Git Bash on Windows to run Make
I have developed a Python template repository that contains a number of makefiles for managing repositories. Among the tasks, it can help with constructing virtual environments (
$ make venv) and installing all pip dependencies. It can optionally launch Jupyter notebooks (
$ make launch). The real power comes from the fact that I can use the same set of commands for the basic management of the Python repositories. It is really very handy on Linux. I do development work on windows and I wanted to be able to use the makefiles there. Unfortunately, there wasn’t an easy way that I really liked. There are options like Cygwin and even WSL for Windows. Both of these options were too heavy to do what I wanted.
Cribbage Strategies Are Explored With New Code And New Methods
This is a rewrite of my previous cribbage article and my article on expected average. It also includes access to completely re-written code. The code is simplified and complete with unit tests. It uses the click library to drive a nice command line/terminal application. This article will assume you are familiar with the rules and the point counting conventions of cribbage. Some of the relevant counting and conventions will be reviewed.
Vector Reflection (2D Derivation)
This notebook will work through the explanation of determining the 2D vector reflection from a surface. I had a problem where I needed to determine the reflected vector from an incident vector in two dimensions. There are a lot of pages out there with good explanations. But there is a lot of seemingly conflicting information that caused me to ask some questions and spawn this article. This source is quite nice. It walks you through the steps and develops a valid relationship:
The problem: We have a 2D circle and we have a ray or line. What are the intersections points between the two, if any?
I have a lot of alerts configured with Google Scholar for various research interests. It’s a very cool concept, setting up a keyword search like
blast fragmentation shockwave and Google sending you a summary email of new research that matches.
Updated on 2021-08-01
New update generated.
Parts of Hashes and Expected Collisions
I am building a documentation system that works using Markdown for the documents and Pandoc to transform the documents to HTML, PDF, etc… It works well and is very easy to use. However there is a problem I have encountered. By default when Pandoc transforms a Markdown file to HTML, it automatically inserts section anchors. In Markdown, an ATX section header could look something like this:
Measurement Uncertainty - How do measurement errors add up when working with areas?
Typically when you measure things, there is a certain amount of error. In everyday life, this is ignored for the most part. I was thinking about uncertainty in measuring areas. Like everyone else, I learned about uncertainty in measurement in high school during science class. It was further reinforced in university in every lab I took. The problem was, it was addressed as a set of rules to memorize and apply that covered the use of significant figures. Most people do not fully explore what this means exactly and consequently have trouble with the concept outside the typical canned responses. I was having trouble understanding why calculating an area with uncertainty was expressed the way it was.
Cleaning Thunderbird Linked to Gmail IMAP Account
I was running backups on my system (Ubuntu 20.04) and realized that my
~/.thunderbird folder was huge. It was about 10 GB in size! That is a bit too much for my liking. I use Gmail as my primary provider. There were a lot of emails (over 50,000) stored on the server. I wanted to organize and move them offline to free up space. Why? Google has always touted that you shouldn’t have to delete anything. Well, I have email going back to 2004, and with Google’s announcement about photos I figured now was the time to free up space.
Installing Python 3 from Source
The purpose of this guide is to allow you to install python 3.x into Ubuntu Linux (or its variants) without affecting the system python installation used for system scripts. We’ll install to:
Change Units in Regression Equation
The purpose of this post is to explore the concept of changing the base units for a regression fit equation. Particularly the coefficients. The work is based on a paper called, “Prediction of Compressive Strength from Other Rock Properties.”
Automating Borg Backup
I wanted to use a proper backup solution to replace my rsync script. I decided to use BorgBackup as it seemed to suit the bill. It is a repository based system that has very strong deduplication algorithms. Essentially, you create a backup repository in a particular path and then backup folders and files to the repository.
Cribbage - Expected Average
As of 2021-07-24, there is a newer article with newer code that you can read here.
Updated on 2021-07-24
There is a newer article with newer code that you can read.
Updated on 2022-01-19
Add more traffic cameras, from Sudbury to North Bay.
Jupyter in a Docker Container
I have been using Jupyter notebooks in a virtual environment for some time now. I would compile the version of Python that I wanted into a local folder that did not require any special permissions. I would then create a virtual environment for Jupyter and proceed to install what I needed. Once completed I created the requirements file. It was fairly easy to update items. A bit time consuming and not fully automated. This didn’t work well for windows. I have to use conda for that platform.
Python - Install from Source - Local
This tutorial is about installing the latest versions of python from the source into the users home folder as opposed to a server-wide install. Normally /opt/python/python-3.6.1 would be the best choice for the installation. In this case, installing locally is fine as well.
Docker - A Cool Option
I decided to upgrade the VPS from Ubuntu 14.04 to 16.04. At the same time, I decided to use Docker containers to compartmentalize the blog. Docker is very cool technology that does for applications what version control has done for code. It allows you to create a container that has all of the dependencies of an application in one immutable container. This means, for me at least, that it is straightforward to upgrade the different pieces.
Install Python and SQLite from Source
I was writing some Python to pull text from pdf files and put them into a sqlite database so that I could perform full text searches for various keywords and phrases. I was able to extract the text and put it in the database. I was using Anaconda on windows to do this fully expecting to be able to do the same on Ubuntu (14.04). I had to replace the sqlite3.dll with the latest one from here because the sqlite3.dll included with Anaconda didn’t have FTS4 or FTS5 enabled. This was as simple as copying the new dll over top of the old one and running this script to verify the changes:
Use Winscp to sync files from Windows to Linux
Recently I upgraded my work laptop to Windows 7. At that time I didn’t want to use the previous sync methods that I have blogged about. I wanted to use something simpler (read easier to install and maintain between different machines). After doing some research I settled on using winscp. Winscp supports folder sync operations through a command line. Winscp takes a simple text file listing the commands that it is to execute. This process can be automated on Windows using batches, one to pull changes and the other to push changes.
Updated Mercurial Batch Pull/Update Python Script
It has been awhile since I last posted. Here is an update to the mercurial push, pull & update scripts I had posted earlier. The code is much better then the original scripts. All of the functionality is wrapped into one script instead of across a few. It should run on Windows without alteration (I’ll get a chance to test it out on Tuesday).
Tolerance Testing - Determining an Appropriate Tolerance
Programming with floating point values leads to numerical round off errors due to the nature of binary numbers. For a more detailed discussion see this article or this one. Basically it boils down to the fact that not all real numbers can be represented by a finite binary sequence. Due to this phenomena comparing floating point values directly is strongly discouraged as the results can be unexpected. Normally, the absolute value of the difference is taken and if it is less than some tolerance value it is accepted as a match.
Team Combinations from a Limited Pool of Players
I had to determine an arrangement of teams from a player pool. Specifically there were 9 players that needed to be organized into fair teams. It seemed straight forward to arrange them into 3 teams of 3 players. The other caveat was that the teams needed to be as fair as possible. Some players were highly skilled while others were not. It wouldn’t be fair to stack the best players on a single team. In order to determine a fair team I had to figure out how many combinations of teams were possible. This would allow me to iterate through all of the combinations and apply a metric to each combination. The combination that produced the minimal value would be the optimal arrangement.
Sync files from Windows to Linux using SSH
Over the weekend I decided to figure out how to sync files between windows based computers and Linux based computers, specifically Ubuntu. On windows I investigated a number of technologies. Finally I settled on cwrsync. The reason for the choice is that I really like rsync. I have a number of scripts that work really well (and are fast) that I use on my Linux boxes on a regular basis. There is rsync available in cygwin but that is far too heavy for simple file synchronization. cwrsync is the best of both worlds. It packages the cygwin dll and rsync binaries in a form that is easy to use on windows.
Speed Up Factor
I watch a lot of Coursera videos and usually view them at 1.25x or 1.5x normal viewing speed. I started thinking about how much that would translate into viewing time.
Sequential Generation from an Index Value
I needed to be able to generate a sequence of letters from a specific index value. Basically, I wanted to loop through a sequence of values and retrieve the corresponding string value. For example, 0 would be A; 4 would be E; 36 would be AK; etc.
Rsync between Windows Folders
Following from the last post, here is an example script that uses cwrsync to sync a network share and another folder. I had to map the network share to a drive before I could use it properly.
Yesterday, everything was working well with my Ubuntu installation. I had to go and mess that up! I thought that I would go and remove packages that I no longer needed. After pruning the files from synaptic everything seemed OK till I restarted the computer. I couldn’t boot into the desktop. I figure I removed something critical. I spent a couple of hours trying to recover.
I have been planning on moving from windows for awhile now. I just hadn’t really analyzed what was keeping me in windows. I finally got around to it and realized that I really only use VB.net and c#. With the mono framework, there should really be nothing to hold me back. So I made the decision to switch my home computer over. I had previously installed Ubuntu on my son’s computer and was very impressed with it.
Python Script to Parse PFSense DHCP Log
I have a captive portal setup on my PFSense which allows my laptops and various other devices to connect through wifi. I was looking at the DHCP logs provided by PFsense the other day and realized that I needed a way to verify the macs that were requesting ip addresses. I put together a python script that parses the log and attempts to match the mac addresses that I know with the ones in the log. Enjoy the code and note that the macs have been changed.
Python Reading Material
A while back I finished a pretty good book on python Python Scripting for Computational Science by Hans Petter Langtangen (link). It was a pretty good introduction to python. I really liked the slant towards the sciences and engineering. The problem sets were good.
I have moved the blog from wordpress to a new hosting provider Cloud-A. They provide a server and I configure it to run. So far the process has been pretty straight forward. I have the blog running on Ubuntu 14.04 and hosted using Nginx. It really didn’t take to long to get things up and running. The longest part was converting my old posts to reStructuredText. I decided to use a so-called static blog generated called Nikola as opposed to another wordpress implementation.
Mercurial and TortoiseHG on Ubuntu
I like Mercurial as a version control system because it is cross-platform (written in python) and is distributed (meaning it doesn’t require a central server to function). I use it on windows quite extensively and was one of the pieces of software that I needed on Linux. The other piece that I needed was TortoiseHG. It is a graphical front end to mercurial and works well.
Mercurial Push/Pull script with status checking
This is a modification to the original script that I published a while back now checks the status (hg status) of the repository before doing anything. If there are uncommitted changes, a message is printed and the repository is ignored in the pull/update mechanism. The check for commit status is also made for pushes as well. It is a very nice improvement to the script.
Mercurial Push/Pull and Update scripts
I like Mercurial as a version control system. It has a number of advantages over more traditional systems such as Subversion. I won’t go into details, they are easy to find on the internet. What I have found with mercurial is that I organize all of my repos under a root directory. I also use TortoiseHG as a graphical client that manages the commits and push/pull cycles. It works well for a single repository. Unfortunately it doesn’t work as well for a large number of repositories, that is it can’t do batch push/pull or updates.
File and Folder Permissions
As I get my Ubuntu system running the way I like I find I am copying files over from my old windows partitions (mp3’s, documents, pictures, etc.). I was looking at the permissions of my pictures - they were set to 777. I didn’t understand why. I think it has to do with the fact that I copied them from a windows ntfs partition. I can understand if it were set to 666, but having an the executable bit set really throw me. I wanted to change my pictures to permissions of 644. I tried running the chmod command in my home folder on my pictures.
Copy Pictures from a Digital Camera and Automatically Rename to Date and Time Taken
Most digital cameras use some sort of naming scheme that leaves a lot to be desired. The names usually consist of something like:
Convert MTS (AVCHD) Files to xvid
I have a Panasonic Lumix camera that generates MTS (AVCHD) movie files. These files are 720p HD files and are really large. I want to store them in a smaller file format without sacrificing quality. Using ffmpeg it is pretty straight forward to convert an MTS (AVCHD) movie file to xvid using ffmpeg. Using the following command will accomplish the goal nicely:
Convert MTS (AVCHD) Files to mkv
Here is a simple shell script that will use ffmpeg to convert mts files to mkv format using the h264 codec to compress them.
Convert MP3s to iPod Audio Book format (M4B)
I had the need to convert a group of mp3 files into a format that was suitable for playing on my iPod. Of course the mp3s could be played directly on the iPod without any trouble. This is great for songs, but an audio book is significantly longer. In my case I have a 40 minute commute each way and most audio books are too long to listen to during a commute. The iPod supports m4b files which are audio book files and they remember where they were stopped so you can resume listening to it after putting the iPod to sleep or listening to your music collection. The audio book format also supports changing the play back speed so it will be read to you much faster.
Configuring MathJax on Ghost
I am going to add MathJax support. In the code injection portion of your settings, add the following code to the header injection mechanism:
Configure Syntax Highlighting on Ghost
For syntax highlighting I am going to use highlight.js because I don’t have to install anything. Simply add the following code to the blog header code injection in the settings:
5-pin Bowling Statistics Calculator
My son plays 5 pin bowling and is a member of YBC Canada. I used to keep track of his average and some statistics using a spreadsheet. I would enter the data after the end of every series of games and then copy the cells down so that the formula were applied and the correct statistics were calculated. This process worked well enough except I started to notice small discrepancies between my calculations and the posted results.
I have switched platforms again. I have moved to the blogging platform.