This last time, I was trying to convert a very important It’s Always Sunny in Philadelphia clip to a reaction gif. Can you believe there is no “thank you” reaction gif from when Dennis reads Charlie’s speech?!. Here’s the gif for your own collections.

The command line functions I found came to get this done came mostly from Funky Cloud Medina’s post on this. I didn’t want to use automator, but just write out a few commands, so here is what I did (on MacOS):

First, install ffmpeg and gifsicle with Homebrew. brew install ffmpeg gifsicle

Next, navigate to the directory where your video is. If you plan on doing this regularly, you can create some permanent directories, but I started with creating two temporary directories for the images and then the final gifs.

mkdir pngs/ gifs/

This will create both gifs and pngs folders in the directory you’re currently in.

Next, we’ll process the movie.

ffmpeg -i Untitled.mov -r 10 pngs/out%04d.png

Untitled.mov is the name of the video file in the directory you’re currently in and outputs each frame to the pngs folder with increment digits. The incrementing digits are so you don’t overwrite everything and end up with just a single picture file.

Next, we’ll use sips, the Scriptable Image Processing System. (Check out man sips from your own CLI for more info!)

sips -s format gif pngs/*.png --out gifs

Almost there! This is processing all the image files into the gifs folder. Let’s now move into the gifs folder with cd: cd gifs

And now we can use gifsicle to merge all the images into a single gif! We’ll do that with the following command:

gifsicle --optimize=3 --delay=10 --loopcount *.gif > ~/Documents/thankyou.gif

And voila! You should have a auto-playing gif file ready for use in all your reactions and emails. Enjoy!

Little did I know at the time that comments from the CLI were already possible! I’ve already begun using comments which has been helpful if I need to look back at my zsh_history file. Here’s an example of a command I would use with my docker services.

docker logs {container_name} --since 5m ;: The container is failing 5 minutes after startup.

Turns out, I don’t need a fancy app to log the input/output of a terminal session! It exists in most unix based systems already and the command is simply this: script. That’s it!

Unlike .zsh_history or .bash_history which only saves the commands you input into the terminal, script will save both your commands and it’s output. Why didn’t I know about this before I started setting up my homelab?!

The next time I want to setup a new service or debug something on a server, I will now make sure I start my session with script {date}-{service_name}.txt and then start writing out commands. As long as I can remember to write my inline comments during the session, looking back and trying to figure out what I was thinking at the time should be a breeze! I can dump these files into my internal wiki as placeholders. Then, ideally, I’ll remove the commands that lead me down a dead end, clean up and expand comments, and easily keep my wiki growing. Ideally being the key word here… I’ll take a dump of history files for the time being.

Either way, I will literally have the output of my brain when doing CLI work in a file. Incredible.

Here’s a quick overview about script, taken from man script.

               SCRIPT(1)

NAME
     script – make typescript of terminal session

SYNOPSIS
     script [-aeFkqr] [-t time] [file [command ...]]
     script -p [-deq] [-T fmt] [file]

DESCRIPTION
     The script utility makes a typescript of everything printed on your terminal.  It is useful for students who need a hardcopy record of an interactive session as proof of an assignment, as the typescript file can be
     printed out later with lpr(1).

     If the argument file is given, script saves all dialogue in file.  If no file name is given, the typescript is saved in the file typescript.

     If the argument command is given, script will run the specified command with an optional argument vector instead of an interactive shell.

Recently, my wife needed help in sharing weekly content with a group of people. The original way this group was sharing content was a PDF export of a Google Doc. From a User Experience perspective, it wasn’t great. Someone would received this long PDF, they would have to scroll to find the date or topic of the next additional_notes, and overall didn’t look great.

While she had developed the Google Sheets and Google Slide system to make everything a bit more legible and navigable, there was still an element of copy and paste from the Sheet to the Slide. She had asked if I knew of any ways to make this process easier. This was a perfect mini-project for Google Apps Scripts!

Below, I’ll share the pieces of the script to explain what is going on. If you know what you’re doing and just want the script, feel free to head to the bottom of the page to see the full script.

Goal

The goal here was pretty straight forward. I wanted to add a button on the Google Sheet that allows the user to create a Slides presentation from a subset of data within the sheet. The data will be date based, as this is what most end users need, and the user will be able to pick from the currently available dates & populated data.

Setup

Getting setup, you need both a Sheet with a few headings and an empty Slide presentation with a few empty text boxes. The empty text boxes will be key to helping us connect the data in the sheet to the correct placement in the slide. Here are two screenshots of what this looks like for my example:

Sheets Setup

Ignoring any design from my screenshots - all credit goes to my much more creative wife - the setup for the sheet is fairly simple. You need various headings in Row A of the sheet which we will be using to reference data. In this tutorial, our headings are Date, Topic 1, Topic 2, Topic 3, Additional Notes. Whether you start on Row 1 or after that doesn’t matter too much.

For the date column, we’ll be formatting our date like this: “February 26, 2024”. You’ll see why in a little bit.

On top of the actual data in the Sheet, the Apps Script is going to live in this document and just push data to the Slide. To access your App Scripts, click Extensions > Apps Scripts. A new tab will open with a blank IDE style interface and an empty myFunction.

Slides Setup

Don’t worry too much about the design for Slides, you can change that later. But the important step is creating the empty text boxes. After you create your text boxes (4 will be used in this tutorial), right click one of them and select “Format Options”. A panel on the left hand-side should slide out. Click the “Alt Text” drop down, and then “Advanced Options”. That little text box is the title for your text box; it is not used in the visual representation of the box, we will just be using it as a reference point.

For ease of this tutorial, make the Title of the text box the same as the Header row from when we set up the Sheet, above. Once you’ve added title to each of the text boxes, let’s head into the code.

onOpen Function

The first function you need for creating a UI change in the Google Sheet is an onOpen function that will setup the UI when the Sheet is open.

1
2
3
4
5
6

function onOpen(e) {
    SpreadsheetApp.getUi()
      .createMenu('Create Presentation')
      .addItem('Create Presentation', 'askDate' )
      .addToUi();
}

In this function (which we’ll later trigger in the script settings), when someone opens up the Sheet, it will create the UI button in the top level menu. Here’s what’s happening:

• SpreadsheetApp.getUi() –> Instantiate class and get available methods for getUi().
• createMenu() –> Creates a menu item called “Create Presentation”.
• addItem() –> Adds an item to that menu that when pressed, calls the askDate function.
• addtoUi() –> Add it! Now people can see and click on it.

askDate Function

This is the main function and a bit long, so I’ll split it up into a few sections.

Section 1

1
2
3
4
5
6
7

  var ui = SpreadsheetApp.getUi();
  var sheet = SpreadsheetApp.getActiveSheet();
  var lastRow = sheet.getLastRow();
  var dates = sheet.getRange(2,2,lastRow,1).getDisplayValues();

  // get today's epoch time for easier calculations
  const epochDate = new Date().getTime()

This first section we’re just getting the sheet ready for analysis. With the dates variable, we’re just looking at the second column. The reason we’re using the getDisplayValues() method is because Google will automatically convert the dates to include time zone, time, etc. We want to keep the date in the same format for a better user experience.

For that last line, we’re getting the current date & time in epoch time so we can run a comparison further down the script. For this use case, we don’t need to include any dates in the past.

Section 2

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

  menuOptions = []
  const re = new RegExp("^[A-Za-z]{3,15}\\s\\d{1,2},\\s\\d{2,4}")
  for (let i = 0; i < dates.length; i++ ) {
    var date = dates[i][0];
    var epochSheetDate = new Date(date).getTime();
    if (date != "") {
      var dateMatch = re.test(date)
      if (dateMatch == true) {
        if (epochDate < epochSheetDate) {
          menuOptions.push(date)
        }
      }
    }
  }

In this section we’re creating an empty array and instantiating a RegExp to ensure we have an actual date in the cell. See Regex101 to learn more about Regex and test different regex syntax.

Next, we dive into a for loop, looping through the dates column of values (which we just called earlier). After attributing each value to the var date variable, we also convert that same value into epoch time (by creating a new Date().getTime()) so that we can compare it with today’s date.

After the variables are setup we need to check that the date isn’t empty; we don’t need any rows where a date hasn’t been assigned to it yet. If we have a non-empty date value, let’s compare it using the regex string. All we’re doing here is asking “Is this date in the format I’m expecting it?” If true, let’s keep the value and continue using it. If not, just ignore it.

So we’ve now found a value that’s in the date format we expect, let’s now take that same value in epoch time (referenced by the epochSheetDate variable) and compare it to today’s epoch time date. If today’s date is less than the value in the sheet, that means the date in the sheet is in the future.

So now we have a date in the correct format and that is at some future date from today. Fantastic! Once we’ve gone through those checks, we’re ready to add the date to the array we created at the top of this section. Push on!

Section 3

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

if (menuOptions.length > 0) {
  let stringList = menuOptions.toString();
  var formattedDates = stringList.replaceAll("2024,", "2024 \n ");
  var response = ui.prompt('Which date would you like to create a presentation for? Please copy and paste the date exactly as you see it in the options below. Options: \n \n'+formattedDates);
  if (response.getSelectedButton() == ui.Button.OK) {
    var dateChosen = response.getResponseText();
    if (dateChosen != "") {
      ui.alert("Woohoo! Let's make a presentation!")
      for(var y = 0; y<dates.length;y++){
        if (dates[y][0] == dateChosen) {
          // Logger.log("Row:"+(y+2));
          var presRow = y+2
          var presData = sheet.getRange(presRow,3,1,4).getValues();
          for (presCopy of presData) {
            var topic1 = presCopy[0];
            var topic2 = presCopy[1];
            var topic3 = presCopy[2];
            var additional_notes = presCopy[3];
          }
          createPresentation(dateChosen, topic1, topic2, topic3, additional_notes)
        }
      }
    } else {
      ui.alert("Ouch. Looks like you entered something incorrectly (i.e. you entered nothing). Try again.")
    }
  }
} else {
  ui.alert("No Dates Available. Check Spreadsheet")
}

This next section might feel long, but it’s really easy to follow, don’t worry. It’s a lot of data organization to ensure we’re giving the users a smooth experience & to make sure when the data hits the Slide, it’s in the right spot.

First things first - let’s only run this if we actually have items in the menuOptions array! No need to give the user an affirmative message when there’s no data to process. If there are no dates available, we call the else of this if, which is the 3rd to last line in this section.

The next few lines are for processing that array of dates we grabbed from the sheet to making them presentable to the user. Without the stringList and formattedDates variables, Google’s modal just shows a wall of text of dates which makes it difficult to parse for the user. By using replaceAll() we remove the comma and insert a new line after every portion of the string that contains 2024 and a comma.

So now, instead of the modal showing:

February 2, 2024,February 3, 2024,February 4,2024... etc

We now see the much easier to read:

February 2, 2024
February 3, 2024
February 4, 2024

After that’s formatted nicely, we can now have the prompt show for the user which asks them what date they would like to process into a Slide presentation. Since we haven’t visited any ui options since the top of this tutorial, this is the modal that will show up after a user clicks “Create Presentation” in the top level menu in Google Sheets.

Ok, so now moving onto line 5, we need to get the text for what the user enters into the modal box. If they don’t enter anything, we should let them know that nothing will be done. That message comes from the second to last else statement in this section. Instead, if they enter some data let’s parse it!

For lines 9 and 10, we’re now looping back through the same set of rows that we did in section 2. The reason we’re re-entering this for loop is because we’ve already exited the loop. So we have to get the values again. Once we have them, let’s match the date that the user entered to the correct row. We’re able to do that by grabbing the index and adding 2 to it.

Now that we have the correct row number, we’re going to shift from getting values from each row and getting them from each column. By calling var presData = sheet.getRange(presRow,3,1,4).getValues(); we are grabbing an array. getRange is looking for the following values, in the following order:

• Row number (which we got from the y+2)
• Column Number (which is Column C in this case)
• Number of Rows (we only need this row)
• Number of Columns (we know how many more columns we need).

Next up, we’re jumping into another for loop, now for all the data we just pulled from the column. This part is just parsing each column’s data into it’s own variable for easier management when we create the presentation. We’ve also grabbed the additional notes column, but we’re not using it right now. This column’s data may be valuable by adding it to the speaker notes of the Slide Presentation. Check out these two resources:

• Working with Speaker Notes
• Notes Properties API

Once that’s done, call the next function: createPresentation()! We’re almost done.

createPresentation Function

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

function createPresentation(dateChosen, topic1, topic2, topic3, additionalNotes) {
    const slides = SlidesApp.openById('ENTER_ID_OF_SLIDE_PRESENTATION');
    let slide = slides.getSlides()[0];
    let elements = slide.getPageElements();
    for (let i = 0; i < elements.length; i++) {
      var titles = elements[i].getTitle()
      if (titles == "Date") {
        var dateText = elements[i].asShape().getText();
        dateText.clear();
        dateText.setText(dateChosen);
      } if (titles == "Topic 1") {
        const topic1Text = elements[i].asShape().getText();
        topic1Text.clear();
        topic1Text.setText(topic1);
      } if (titles == "Topic 2") {
        const topic2Text = elements[i].asShape().getText();
        topic2Text.clear();
        topic2Text.setText(topic2);
      } if (titles == "Topic 3") {
        const topic3Text = elements[i].asShape().getText();
        topic3Text.clear();
        topic3Text.setText(topic3);
      }
    }
}

This function may be the shortest to explain. All we are doing is taking those variables from the previous function (the variables that contain each topic’s text value) and doing the following:

1. Opening the Presentation (line 2)
2. Getting the first slide (line 3)
3. Grabbing all the page elements (line 4)
4. Looping through the page elements & grabbing all the titles that we set during setup (line 5 & 6)
5. Then we compare the title of the Slide text box to the expected string (Topic 1-3), and if we get a match, retrieve the text of the box as a variable, clear the text, and input our new text with the variables we created in the previous function.

And just like that, you should have all your boxes filled in with the data you input to the Google Sheet!

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87

function onOpen(e) {
    SpreadsheetApp.getUi()
      .createMenu('Create Presentation')
      .addItem('Create Presentation', 'askDate' )
      .addToUi();
}

function askDate() {
  var ui = SpreadsheetApp.getUi();
  var sheet = SpreadsheetApp.getActiveSheet();
  var lastRow = sheet.getLastRow();
  var dates = sheet.getRange(2,2,lastRow,1).getDisplayValues();

  // get today's epoch time for easier calculations
  const epochDate = new Date().getTime()

  menuOptions = []
  const re = new RegExp("^[A-Za-z]{3,15}\\s\\d{1,2},\\s\\d{2,4}")
  for (let i = 0; i < dates.length; i++ ) {
    var date = dates[i][0];
    var epochSheetDate = new Date(date).getTime();
    if (date != "") {
      var dateMatch = re.test(date)
      if (dateMatch == true) {
        if (epochDate < epochSheetDate) {
          menuOptions.push(date)
        }
      }
    }
  }

  if (menuOptions.length > 0) {
    let stringList = menuOptions.toString();
    var stringlist = stringList.replaceAll("2024,", "2024 \n ");
    var response = ui.prompt('Which date would you like to create a presentation for? Please copy and paste the date exactly as you see it in the options below. Options: \n \n'+stringlist);
    if (response.getSelectedButton() == ui.Button.OK) {
      var dateChosen = response.getResponseText();
      if (dateChosen != "") {
        ui.alert("Woohoo! Let's make a presentation!")
        for(var y = 0; y<dates.length;y++){
          if (dates[y][0] == dateChosen) {
            // Logger.log("Row:"+(y+2));
            var presRow = y+2
            var presData = sheet.getRange(presRow,3,1,4).getValues();
            for (presCopy of presData) {
              var topic1 = presCopy[0];
              var topic2 = presCopy[1];
              var topic3 = presCopy[2];
              var additional_notes = presCopy[3];
            }
            createPresentation(dateChosen, topic1, topic2, topic3, additionalNotes)
          }
      }
    } else {
        ui.alert("Ouch. Looks like you entered something incorrectly (i.e. you entered nothing). Try again.")
      }
  }
} else {
    ui.alert("No Dates Available. Check Spreadsheet")
  }
}

function createPresentation(dateChosen, topic1, topic2, topic3, additionalNotes) {
    const slides = SlidesApp.openById('ENTER_ID_OF_SLIDE_PRESENTATION');
    let slide = slides.getSlides()[0];
    let elements = slide.getPageElements();
    for (let i = 0; i < elements.length; i++) {
      var titles = elements[i].getTitle()
      if (titles == "Date") {
        var dateText = elements[i].asShape().getText();
        dateText.clear();
        dateText.setText(dateChosen);
      } if (titles == "topic1") {
        const topic1Text = elements[i].asShape().getText();
        topic1Text.clear();
        topic1Text.setText(topic1);
      } if (titles == "topic2") {
        const topic2Text = elements[i].asShape().getText();
        topic2Text.clear();
        topic2Text.setText(topic2);
      } if (titles == "topic3") {
        const topic3Text = elements[i].asShape().getText();
        topic3Text.clear();
        topic3Text.setText(topic3);
      }
    }
}

The last two times I’ve run apt update && apt upgrade -y on my web server and OpenResty has an update to push out, I will get some errors including Openresty is not setup correctly. There are a few other errors in the systemctl status openresty output such as process 223478 [nginx] is still running. I might be slightly paraphrasing the errors, but that’s roughly what I’m finding.

Like any debug session, I make sure nginx is disabling and not running (systemctl disable and systemctl stop), which I can confirm. Now, Openresty does use Nginx under the hood, so that errors makes me think it’s just conflicting services trying to run on top of each other.

The weirder part is when it warns me that OpenResty is not setup correctly. I didn’t change anything… so what is the update/upgrade trying to setup?

Next, I’ll look through my config files (i.e. nginx.conf) - no changes there either and nothing that stands out as out of the ordinary.

One of my bad habits is that when doing this sort of debugging and running something like journalctl --since 21:45:00, I’ll look at the logs more closely with each try and fail. I should just look more closely from the beginning! But I digress.

Since the nginx process and openresty setup errors are the most plentiful but yield nothing, I’ll look back through the logs for the single lines that I miss on my first few passes. That’s when I see it, buried between the other errors. A single line that says my pid file can’t be found.

My pid file for OpenResty and Nginx is stored in my /run directory, but for whatever reason, whenever OpenResty pushes out an update, it overwrites my systemd file and starts looking for the pid file under /usr/local/openresty/nginx/logs/nginx.pid. Why the file would be located under a logs directory is still beyond me, but a quick update to the systemd file, a systemctl daemon-reload and then systemctl start openresty and all my public facing services are back in action.

After I fixed it for the second time, I realized I hadn’t written this down in my documentation. So I’ve now recorded it and thought I’d share it here in case it helps anyone else. I also did a bit of research after and found that even in this Digital Ocean post, it has OpenResty/Nginx’s PID file in a different location too. I couldn’t find OpenResty’s documentation on this yet, but Nginx’s Official Docs have the PID file in the same location as I am storing it. Perhaps this is just a mis-config between nginx coming natively on my linux distro and then OpenResty being installed on top of it?

Either way - problem solved, and documentation captured. That’s a win for my day.

In order to greatly reduce the number of nodes needed in the single recipe, I created a list of dictionaries in python (with one of the values being another list!) and a quick for loop to grab all the appropriate elements from the various dictionaries. I had thankfully already copied all the JS nodes to a single file for easy reference and retrieval. That one file had over 8,000 lines, and I needed to replace all the javascript elements with python.

Here’s one node of Javascript that I was working with:

// @param input fields supplied in the recipe step, as an object
// @return object matching the output schema
// Eg: Code for returning time zone for an IP address

exports.main = async ({ user_email }) => {
  let all_domains = [
 'domain_one',
 'domain_two',
 'domain_three',
 'domain_four',
 'domain_five',
 'domain_six',
]

  let user_domain = user_email.slice(user_email.indexOf('@'));
  let domain_found = all_domains.indexOf(user_domain) > -1;
  return { domain_found };
}

Most nodes were very similar, all except that their all_domains array had anywhere from 1 to 500+ elements. Thanks to the neovim plugin Telescope Cmdline by Jon Arrien, you can very easily see your past history of Neovim commands. I’ve gone back through and pulled out a few special ones that helped and that I was very surprised how well they worked. I don’t claim that this was the fastest or best way to clean up a large file with repeating functions, but I definitely learned a bunch about searching-and-replacing in Neovim, and any future needs for this will be way faster! One thing that has been really helpful with Neovim is that as you run the first portion of all the commands below, Neovim should highlight what it finds. This is really helpful for real-time debugging of your regex.

Commands and Quick Explanations

Command: :g/\//+2/d

Explanation: The g in the command stands for global (hint, run :h :g in vim!) and will search across the entire buffer you have open. In this command I’m looking for all Javascript single-line comments. Since the forward slash is a special character for neovim, you need to escape it with a backwards slash, hence the odd looking \/. The second forward slash is the next parameter neovim is looking for, and by adding +{num}, you can search for the pattern plus a number of lines. Closing out the command (again, next parameter comes after the forward slash) with d for delete will then delete everything with you searched for, plus the additional number of lines.

Bonus: if you wanted to make sure that you only pulled lines that began with a comment, you’d add a carrot to the beginning of the command. :g/^\//+2/d

Command: :%s/let all_domains = /"domains" : /g

Explanation: Moving on from g to s, the s here stands for substitute. There’s one big difference in using g and s, though. While g will search globally, a singular s will only search on your current line. To search the entire file/buffer, you need to include the percent character before the s. The rest of this is fairly straight forward, I am searching for the declaration of a javascript variable and then substituting it with a python dictionary key. Since both javascript and python declare arrays with square brackets, I wanted to make sure I kept that after the substitution. Lastly, unless you only want the subsitute to change the first occurrence of what it finds, you’ll need to close out the command with the g flag. Neovim’s docs say:

[g] Replace all occurrences in the line.  Without this argument,
 replacement occurs only for the first occurrence in each line.  If the
 'gdefault' option is on, this flag is on by default and the [g]
 argument switches it off.

Command: :%s/^exports.main.*/{ "uuid" :/g

Explanation: Really similar to the above command, but in this case, there are no characters I needed to save post-substitution. We already know what %s does, so the pattern I’m looking for is all lines that start with exports.main. Since I don’t need anything after that start of the line, I included .* which looks for any character until the end of the line. After the pattern I write out what I need to replace it with, which is the first portion of the dictionaries I need. Finally, we close it out with a global change so it changes it everywhere.

Command: :%s/return.*/], Explanation: Finally, I used this one to close out the domains list/array that I’m using.

Bonus Command: %s/"props": \[\(.*\)\]\,/"props" : \1,/g

Explanation: I won’t go over all the characters and regex commands as I’m reusing a lot of them from the above commands. What is different here is that I’ve lumped the wild card in parenthesis like so: (.*\). Now, I can call back what is pulled by using \1. The command above is looking for a dictionary item with a list as the value. I didn’t need the list, just a string, since all values in my list of dictionaries are single values. By searching for "props": [ string ], and using the \1, I’m able to keep the string intact and the result is "props": string,.

This was the milestone command to learn for me and helped unlock so much potential in my brain for other use cases. For all the other commands I’ve gone over above, I’m fairly simply just replacing characters I no longer need that exist either before or after characters I do need. Now that I can keep strings intact for substitutions, I bet I could revisit all the above commands and make them even more efficient.

What I’d like to learn next is if I can use an “or” statement in the regex. Let’s say I have the same situation as above - a list of dictionaries with some dict values being a list. Well, if the lists are all formatted differently, some on new lines, others on a single line, can I select both in a single command? Something like: :%s/"domains": \[\(.* OR \n\)/.

That will be a post for another time! Let me know on Mastodon if you have any other Neovim search and replace tips and tricks that have become invaluable to your workflow. I’m always amazed and impressed with this community and how differently people are able to use the same tool.

By the way, here’s the final output of running the above commands on the sample code above, converted to Python.

mappings = {
  'uuid': "1234-1234-1234-1234",
  'domains': [
     'domain_one',
     'domain_two',
     'domain_three',
     'domain_four',
     'domain_five',
     'domain_six',
    ],
  'props': 'property-mapping',
}

Resources

Here are some various links of resources that I found helpful when learning about the above commands. I’m assuming you’ve already read through anything in Neovim’s Helpdocs (:h), but sometimes you need a different take and explanation.

• Vim Search, Find & Replace - Reddit
• Search Across Multiple Lines - Vim Tips Wiki
• Vim Search and Replace with Examples - The Valuable Dev
• Removing Specific Characters in Vim - Stack Overflow

Right now, I am using a whopping 77 plugins. That’s mostly because I’m not great at cleaning out old plugins I don’t use as much anymore. Since I am client facing, a lot of them are around making my Markdown notes easier to manage, and Python plugins to help with scripting and file management in the terminal. This poll shows 10-25 plugins being the most common amount.

How many plugins in your config. List your favorites!
byu/Bamseg inneovim

Once I got my dotfiles to my server environment, I got to work and got rid of all the superfluous plugins like additional color schemes, todo-trouble, or others that I considered nice-to-have. After all, across all my VMs I’m mostly editing yaml, json, and python scripts and I’m not in there for a very long time. If I was a real pro, I would just use out-of-the-box Vim, but I really like the speed some key maps and plugins have given me.

You can check out my git branch here. If you decide to clone it, make sure you symlink .config folder from a ~/.dotfiles/ directory to ~/.config/. I’ve been using stow, but any symlink tool will work.

If you have a neovim config that you’d like to “minimize”, see some steps below. For any dotfile config - Neovim, tmux, zsh - I like to make these changes in a clean environment. I want to replicate as closely as possible what a fresh install and setup would be like. Here’s how I do it:

1. Navigate to the machine/vm/server that you want to work on. For me, this VM doesn’t have Neovim installed yet.
2. Install Neovim
3. Clone your main repository to either your .config or your .dotfiles directory. And run a few commands:

• git branch {branch-name}
• git checkout {branch-name}

4. You’ll now be in your branch, but everything will look the same. Start removing directories or files you don’t need.
5. Navigate to your plugins folder or init.lua that lists all your plugins and start hacking and slashing.
6. Once you’ve saved all those files and you can open and close neovim without any plugin errors, you’re ready to go.
7. Navigate back to the root git directory and run a few more commands:

• git add .
• git commit -m {Insert a detailed commit message of what you changed and updated.}
• git push {remote-name} {branch-name}

8. And that’s it! Go check your Github/Gitlab and check your branches and commits. You should see everything there! Congrats.

For that last command, you can always check what your remote repo’s push name is by running git remote -v. Most often, the name will be something like origin or main. If you saw my post on having multiple remote repos, you may something like “all”. Just make sure to check before pushing!

If you’re following up on this from my first post, you’ve already added your AppDaemon script and confirmed that the AppDaemon logs don’t show any errors. Now is the true test if it’s working: do you have the new sensors in HomeAssistant?!

Dashboard Overview & Dependencies

Now that we have the correct entities, lets take a look at what we’re working with. Full disclosure that once I got this working, I haven’t really revisited it, refactored it, or made any improvements. I’m sure you’ll find ways to use less YAML, but I wanted to get this out there sooner than later!

I’ve included the code for all the cards at the bottom of this post. You can find them here. This dashboard is pretty simple, it brings in almost all of the sensors we created in the first post and organizes them in an as-pleasant-as-possible view. I’m definitely not a designer, so some of the colors could be worked on…

Sensors in the dashboard:

• Overview Card:
  • sensor.bird_common_name (only used to generate the picture)
  • camera.birdnet_flickr
  • sensor.bird_common_name
  • sensor.bird_science_name
• Data Card:
  • sensor.bird_time_seen
  • sensor.bird_confidence
  • sensor.bird_last_seen
• Weather Card:
  • weather.pirateweather
• Description Card:
  • sensor.birdnet_wiki

There are also two HomeAssistant dashboard dependencies that you’ll need for this dashboard:

• Custom Weather Card
• Custom Button Card
• Custom Card Mod

Now that we’ve got that squared away, let’s jump into each card.

Overview Card

You’ll notice from the few dependencies listed above, that I use this button card. A lot. RomRider did a fantastic job of adding in a ton of flexibility into the card. For the overview card, we’re taking one of the entities, in this case, bird_common_name and attaching the Flickr picture/sensor to it. Then, on the right, I’m displaying the name and common name. Here’s the overview card’s yaml.

The tricky or tedious part of this card is making sure most of the card’s attributes (icon, name, state, label) are set to false. Another option for the Common and Scientific name on the right is to use a markdown card. I couldn’t get the formatting just right when using that card and some grid-card tricks, so I opted to reuse the super flexible button-card.

Pay attention to the styles for the image (lines 24-40). Those are what keep the border around the image along with the image a certain height and width so that it looks proportional on the page.

One additional thing I have been toying with but hadn’t finalized was messing with [img_cell][border]. By adding some javascript in that section (see other cards), you could change the color of the border based on another parameter. Perhaps you look at the state of sensor.bird_common_name and if there is the name of a color in there, that’s the border’s color. Feel free to get crazy and creative with this!

type: horizontal-stack
cards:
  - type: custom:button-card
    entity: sensor.bird_common_name
    triggers_update: all
    show_name: false
    show_icon: false
    show_state: false
    show_label: false
    styles:
      card:
        - background: transparent
        - border: none
        - width: 215px
        - height: 175px
    custom_fields:
      picture:
        card:
          type: custom:button-card
          entity: camera.birdnet_flickr
          show_entity_picture: true
          show_name: false
          show_icon: false
          styles:
            card:
              - height: 100%
              - width: 100%
              - padding: 0px 15px 0px 15px
              - border-radius: 3px 3px 15px 3px
              - border: none
              - background: transparent
              - overflow: visible
            img_cell:
              - width: 180px
              - height: 160px
              - border-radius: 69%
              - border: 3px solid grey
            entity_picture:
              - width: 215px
              - height: 100%
  - type: vertical-stack
    cards:
      - type: custom:button-card
        entity: sensor.bird_common_name
        show_entity_picture: true
        show_state: true
        show_name: false
        show_icon: false
        styles:
          card:
            - background: transparent
            - border: none
            - margin-top: 35px
            - font-size: 25px
            - width: auto
      - type: custom:button-card
        entity: sensor.bird_science_name
        show_entity_picture: true
        show_state: true
        show_name: false
        show_icon: false
        styles:
          card:
            - background: transparent
            - border: black
            - width: auto

Data Card

This card is fairly straight forward in that it’s showing 3 key data points: the time of detection, detection confidence, how long ago it was seen. This could be fairly redundant since we already have the time of detection, but when you’re just quickly glancing at the dashboard, minutes ago is much faster brain processing than comparing the timestamp and the current time.

You’ve likely picked up by now that in the previous post, we never sent a payload to create the sensor.bird_last_seen entity. Here’s how you can do it.

Creating Bird Last Seen Entity

When I first set out to create this sensor, I was messing with jinja templates for timestamp, datetime, strptime, and more. Here are a few code blocks I saved in my notes in case I went too far down the wrong path. Here are a few of them.

{{ now() - state_attr(sensor.bird_time_seen, 'last_triggered') > timedelta(hours=24) }}

or:

{% set bird = strptime(states('sensor.bird_time_seen'), "%H:%M:%S") %}
{{ relative_time(bird) }}

The thing is, HomeAssistant has already implemented this really neat feature for calculating time, especially from when something was last updated. This function is called relative_time. Having something like this allows you set automations to run after a specific amount of time has passed since the last time a sensor or entity was updated.

The issue I faced with relative time has to do with the sensors I created from my AppDaemon script. Relative time expects a date and time. I was only passing the time. In Home Assitant if you use the Developer Tools > Template to test relative time out on the sensor.bird_time_seen sensor, you’ll get a result of 126 years… That’s because without a date, Home Assistant defaults the date to 1900-01-01. The full relative_time return is 1900-01-01 15:15:15.

We could go back and set the sensors to include both date and time, but I prefer them separate so that I can use them in different places. For this dashboard, the day is always today, so having the date felt redundant. To create a new sensor using the relative_time function, you’ll need to edit your configuration.yaml.

Once you’re editing your config file, add the following:

template:
# Bird Time Last Seen
  - sensor:
      - name: "Bird Last Seen"
        state: >
          {% set birdseen = (states('sensor.bird_date_seen')+' '+states('sensor.bird_time_seen')) %}
          {% set bird = relative_time(strptime(birdseen, '%Y-%m-%d %H:%M:%S')) %}
          {{ bird }}          

What this does is uses Home Assistant’s templating functionality and creates a new sensor called “Bird Last Seen”. The default sensor. name will be sensor.bird_last_seen.

To configure the state of that sensor, we first set a variable called birdseen. To this variable we are assigning the concatenated values of bird_date_seen, a single whitespace, and bird_time_seen. We’re choosing this format because that is the format that relative_time returned before when we tried using it without a date.

As a quick experiment, take the templating code under the state: > parameter above and throw it into Developer Tools > Template. Do you get 126 years? Or something more realistic? If something more realistic, amazing!

We’re almost there! Here’s what you should see in HomeAssistant if the sensor was created correctly.

type: horizontal-stack
cards:
  - type: custom:button-card
    entity: sensor.bird_time_seen
    show_state: true
    show_icon: true
    show_name: false
    icon: mdi:clock-outline
    color: darkgrey
    styles:
      card:
        - border: none
        - background: transparent
  - type: custom:button-card
    entity: sensor.bird_confidence
    show_state: true
    show_icon: true
    show_name: false
    icon: mdi:check-circle
    styles:
      card:
        - border: none
        - background: transparent
      icon:
        - color: |
            [[[
              if (states['sensor.bird_confidence'].state > 80 )
                return "green";
              return "lightblue";
            ]]]            
  - type: custom:button-card
    entity: sensor.bird_last_seen
    show_state: true
    show_icon: true
    show_name: false
    icon: mdi:timer-refresh-outline
    styles:
      card:
        - border: none
        - background: transparent
      icon:
        - color: |
            [[[
               var y = states['sensor.bird_last_seen'].state;
               let x = y.slice(0, 2);
               var e = Number(x);
               if (e < 5) return '#ff6969';
               if (e < 10) return '#ffdf87';
               if (e < 15) return '#d9d76f';
               if (e < 20) return '#fcc2ea';
               else return '#ccccc8';
            ]]]            

Weather Card

This doesn’t need a lot of explaining or instructions. It is just the standard weather card! Here’s the YAML, none of the less, so you know what I toggled on/off. I’m using Pirate Weather Integration as my data source.

type: custom:weather-card
entity: weather.pirateweather
forecast: false
hourly_forecast: false
name: null
details: true
current: true
number_of_forecasts: '5'

Description Card

Finally, we reach the bottom of the dashboard: the description card. This one is also really straightforward. We’re just using a standard markdown card and taking the description sensor we created using Wikipedia’s API and making that the main content of the card.

Other than setting the theme, the only other small changes are removing the border and increasing from the default font size. We’ll use Thomas Loven’s famous Card Mod for that.

type: markdown
content: '{{ state_attr(''sensor.birdnet_wiki'',''description'')}}'
theme: Catppuccin Mocha
card_mod:
  style: |
    ha-card.type-markdown {
      border: none;
    }
    ha-markdown {
      font-size: 16px;
    }    

Conclusion

And that’s all there is to it! I say that flippantly, but I know that it can seem like there’s a lot of setup. Everything I did here evolved out of other people’s projects and dashboards on Reddit or the invaluable HomeAssistant Community

Please feel free to reach out to me on Mastodon if you have any questions or get stuck anywhere!

Full Dashboard YAML

- theme: Catppuccin Macchiato
    title: BirdNet-Dashboard
    path: birdnet-dashboard
    icon: mdi:bird
    type: custom:vertical-layout
    badges: []
    cards:
      - type: horizontal-stack
        cards:
          - type: custom:button-card
            entity: sensor.bird_common_name
            triggers_update: all
            show_name: false
            show_icon: false
            show_state: false
            show_label: false
            styles:
              card:
                - background: transparent
                - border: none
                - width: 215px
                - height: 175px
            custom_fields:
              picture:
                card:
                  type: custom:button-card
                  entity: camera.birdnet_flickr
                  show_entity_picture: true
                  show_name: false
                  show_icon: false
                  styles:
                    card:
                      - height: 100%
                      - width: 100%
                      - padding: 0px 15px 0px 15px
                      - border-radius: 3px 3px 15px 3px
                      - border: none
                      - background: transparent
                      - overflow: visible
                    img_cell:
                      - width: 180px
                      - height: 160px
                      - border-radius: 69%
                      - border: 3px solid grey
                    entity_picture:
                      - width: 215px
                      - height: 100%
          - type: vertical-stack
            cards:
              - type: custom:button-card
                entity: sensor.bird_common_name
                show_entity_picture: true
                show_state: true
                show_name: false
                show_icon: false
                styles:
                  card:
                    - background: transparent
                    - border: none
                    - margin-top: 35px
                    - font-size: 25px
                    - width: auto
              - type: custom:button-card
                entity: sensor.bird_science_name
                show_entity_picture: true
                show_state: true
                show_name: false
                show_icon: false
                styles:
                  card:
                    - background: transparent
                    - border: black
                    - width: auto
      - type: horizontal-stack
        cards:
          - type: custom:button-card
            entity: sensor.bird_time_seen
            show_state: true
            show_icon: true
            show_name: false
            icon: mdi:clock-outline
            color: darkgrey
            styles:
              card:
                - border: none
                - background: transparent
          - type: custom:button-card
            entity: sensor.bird_confidence
            show_state: true
            show_icon: true
            show_name: false
            icon: mdi:check-circle
            styles:
              card:
                - border: none
                - background: transparent
              icon:
                - color: |
                    [[[
                      if (states['sensor.bird_confidence'].state > 80 )
                        return "green";
                      return "lightblue";
                    ]]]                    
          - type: custom:button-card
            entity: sensor.bird_last_seen
            show_state: true
            show_icon: true
            show_name: false
            icon: mdi:timer-refresh-outline
            styles:
              card:
                - border: none
                - background: transparent
              icon:
                - color: |
                    [[[
                       var y = states['sensor.bird_last_seen'].state;
                       let x = y.slice(0, 2);
                       var e = Number(x);
                       if (e < 5) return '#ff6969';
                       if (e < 10) return '#ffdf87';
                       if (e < 15) return '#d9d76f';
                       if (e < 20) return '#fcc2ea';
                       else return '#ccccc8';
                    ]]]                    
      - type: custom:weather-card
        entity: weather.pirateweather
        forecast: false
        hourly_forecast: false
        name: null
        details: true
        current: true
      - type: markdown
        content: '{{ state_attr(''sensor.birdnet_wiki'',''description'')}}'
        theme: Catppuccin Mocha
        card_mod:
          style: |
            ha-card.type-markdown {
              border: none;
            }
            ha-markdown {
              font-size: 16px;
            }            

Update: 10/11/2023. A huge thanks to Mastodon User e_mobile2014 who found a broken link in this guide and pointed out that I never explained how to get the mqtt sensors into HomeAssistant!

What you will need

• BirdNET-Pi
• HomeAssistant
• AppDaemon
• MQTT Broker (I use Mosquitto)

Background

In early 2023, at the height of the Raspberry Pi shortage I felt like a king with an extra Rpi laying around, not being used. I’m a big fan of any sort of passive intake of information and had been looking around for various citizen science-style projects that can capture information from the world around me. Since I’m already running an ADS-B antenna with Flight Aware, I figured this next project would deal with radio waves/transmissions. Instead, to my amazement, I discovered BirdNET-Pi!

What is BirdNET-Pi?

In case you didn’t click the links above, BirdNET-Pi is an app built specifically made for Rapsberry Pi devices, that builds off the BirdNET Framework. BirdNET is one of the most advanced acoustic monitoring tools available for passively monitoring bird diversity populations. Where BirdNET-Pi takes it to the next level is the ability to setup an SBC - hopefully enclosed in a waterproof space! - and monitor birds in your local environment over time.

I think this project is beyond neat. It runs a bit slow on a Raspberry Pi 3, but overall it runs smoothly. I was even able to contribute a PR to the project in April when I noticed a bug in the platform after a hard reset of my Pi.

BirdNET-PI Notification Setup - MQTT

Once you have BirdNET-Pi up and running, you’ll need to head over to the Settings and setup the correct MQTT payloads. Here are the possible variables you can pass in an MQTT payload:

• $sciname: Scientific Name
• $comname: Common Name
• $confidence: Confidence Score
• $confidencepct: Confidence Score as a percentage (eg. 0.91 => 91)
• $listenurl: A link to the detection
• $date: Date
• $time: Time
• $week: Week
• $latitude: Latitude
• $longitude: Longitude
• $cutoff: Minimum Confidence set in “Advanced Settings”
• $sens: Sigmoid Sensitivity set in “Advanced Settings”
• $overlap: Overlap set in “Advanced Settings”
• $flickrimage: A preview image of the detected species from Flickr. Set your API key below.

For our purposes, we will only be using $comname, $sciname, $date, $time, $week, and $confidence. However, this entire process is extremely customizable, which you’ll learn more about in the AppDaemon section. Please expand on it and include information that is pertinent to your own uses.

Here is how I’ve setup my MQTT payload from BirdNET-Pi Settings:

Here it is in text form:

Notification Title: $comname,
Notification Body: $sciname, $date, $time, $week, $confidence
[ ] Notify each new infrequent species detection (< 5 visits per week)
[ ] Notify each new species first detection of the day
[X] Notify each new detection
[X] Send weekly report
Minimum time between notifications of the same species (sec): 5

To test my MQTT notifications, I use the iOS client “MQTTool”. After signing up, head to “Subscribe” and type birdnet as the topic and then click Subscribe. If everything is setup correctly and there are birds being recorded by the BirdNET-Pi’s microphone, you should start seeing those detections in the MQTTool app. If so, fantastic news! Let’s move onto AppDaemon.

AppDaemon Script

Now that we have the Pi communicating via MQTT, it’s time to get that information into HomeAssistant. I’ve shared the full script at the bottom of this page, but let’s jump into each section. This is not a full tutorial of how to use AppDaemon, but it may help fill in any knowledge gaps with the system.

Imports

First, we’re going to import time and requests. We’re going to use time as a backup to the $time component in the payload. This can be helpful to see if there delays, or if BirdNET-Pi stopped detecting. We’re then going to use requests to pull from Wikipedia’s API and grab a description for our HomeAssistant Dashboard.

Class Definition

To start any AppDaemon app, you need to include a Class that is defined in the apps.yaml file. This is also where we initialize and define the various items that will be used in the remainder of the script.

class birdnet(adbase.ADBase):
    def initialize(self):
        self.hassapi = self.get_plugin_api("HASS")
        self.adapi = self.get_ad_api()
        self.mqttapi = self.get_plugin_api("MQTT")
        self.birdnet_mqtt = "birdnet"
        self.mqttapi.listen_event(
            self.birdnet_message, "MQTT_MESSAGE", topic=self.birdnet_mqtt
        )

For this script, we need to use a lot of the AppDaemon APIs across more than just HomeAssistant, so we’re going to be using ADBase. By using that, we can initialize the various APIs, which we do in the next 3 lines. In these 3 lines we need to get access to HomeAssistant’s APIs, AppDaemon’s APIs, and MQTT APIs - the first and third items are plugins of AppDaemon, and AppDaemon is… well… AppDaemon! Here are a few reference docs:

• MQTT AppDaemon API
• HomeAssistant AppDaemon API
• AppDaemon API

These will indispensable to you as you leverage AppDaemon and expand this little script.

Once we have access to that, we need to setup the main topic for MQTT from BirdNET-Pi and finally, what event we are listening for that will trigger the functions in the rest of the script. self.birdnet_mqtt = "birdnet" is the definition for the MQTT topic. Let’s breakdown the last line of the class.

Here’s a breakdown of each of the items in that last line. You can find the official documentation here.

• self.mqttapi.listen_event - this is what we use in AppDaemon to listen for an MQTT event in order to trigger a function.
• self.birdnet_message - the name of the function you’d like to trigger
• "MQTT_MESSAGE" - The default event in AppDaemon’s MQTT API plugin. This is used because MQTT doesn’t keep a state in this plugin.
• topic=self.birdnet_mqtt - The topic that will be received to trigger the function. Defined on the previous line.

In other words, what we are telling AppDaemon is the following: “When AppDaemon’s MQTT API plugin receives a message with the topic of ‘birdnet’, run the function birdnet_message.”

birdnet_message Function

Part 1: Variables Management

Now we get into our first function of the script. The first portion of this script is splitting up the payload that we defined from the BirdNET-Pi UI into individual variables that we can better manage later on. If you test this script out by adding print() statements at various points, you’ll notice that the payload is received with the following json formatting:

{
 "payload": {
  "data": "data"
 }
}

As such, we need to look inside the payload to begin grabbing the data. The pre_split variable is now just looking at the data inside the payload and the rest of the variables take all the date into the payload, split it by the comma, and then grab the string by their index. If you remember what we did above above, you’ll see that we have the various BirdNET information at each of the indexes in the AppDaemon script - 0 through 5.

Part 2: Re-Publishing MQTT Payloads

This next section is shooting all the variables we just defined back via MQTT. The reason why we do it this way is because we need HomeAssistant to grab each of these variables as individual sensors. BirdNET doesn’t give us that capability - it’s a single message with all the information in one. Here is the documentation from AppDaemon on mqtt_publish. Later on, I’ll show you how to ensure that HomeAssistant takes those topic payloads and adds them as entities in your HA setup.

Part 3: Wikipedia Sensor

The next eight lines are a fairly straightforward API call to Wikipedia. We start out by passing the science_name into the URL. The rest of the flags that we are passing into the URL comes from Wikipedia’s Docs. url = f"https://en.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&exintro&explaintext&redirects=1&titles={science_name}"

Once that’s done we call it with response.get(url) and format it with response.json(). Wikipedia returns the json payload with the top level of query (which was our action in the url ;) ), and we’re looking for the value within that query.

All that’s left is to take that query value and push it to HomeAssistant! We can do that with the self.hassapi.set_state function. Within the parenthesis we define the name of the sensor (sensor.birdnet_wiki), what it’s state should be (on), and any attributes associated with the entity. Since we can’t assign a long description to the basic status of the entity, we’re adding an attribute with the key of description and the value will be the wikipedia description garnered from the API call.

        url = f"https://en.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&exintro&explaintext&redirects=1&titles={science_name}"
        response = requests.get(url)
        response = response.json()

        for value in response['query']['pages']:
            wiki_desc = response['query']['pages'][value]['extract']
            self.hassapi.set_state("sensor.birdnet_wiki", state='on',
                                   attributes = {"description": wiki_desc})

Part 4: Generate Picture for Detection (Optional)

This part is optional but I noticed that BirdNET-Pi was already grabbing a Flickr Picture for it’s front end, so I took the code from the BirdNET code base and adjusted it a bit for my needs. This will work very similarly to the Wikipedia API call, the main difference here being that you need an API key for Flickr. You can find more information here.

Given Flickr’s fairly robust API, by passing in the detected bird’s common name, we get amazing results from the community of various pictures of the same species of bird. Ever since I’ve set this up, I’ve not seen a mislabeled picture in my dashboard!

The most confusion portion of this section is the image_url as you’ll notice a bunch of data["value"] strings at various portions of the URL. The short answer to this is in the previous line with the data variable. A successful query has Flickr returning a large payload of information. We’re specifically using this Flickr API endpoint. While you can pass a lot of variables for your needs, if you scroll down, you can see that the example response contains multiple photos in a single response. We’re passing per_page=5 to limit some of those response items. Left out of that response, though, is a one-stop-shop for a URL to that photo. Thankfully, Flickr can help us put together a URL from the data in the response.

With this response, we now have the variables we need to construct the URL to actually render the image. Those variables are: Farm ID, Server ID, ID and Secret. I haven’t yet looked into why we need “farm” when the official documentation doesn’t state anything about it.

Almost there! We now do the same as we did with the Wikipedia API response. We create a sensor in HomeAssistant! We’re calling this sensor sensor.birdpic, ensuring the state=on, and giving it the attributes of the image_url as garnered from Flickr.

        headers = {'User-Agent': 'Python_Flickr/1.0'}
        flickr_api = "enter_your_api_key"
        flickr_url = f"https://www.flickr.com/services/rest/?method=flickr.photos.search&api_key={flickr_api}&text={common_name} bird&sort=relevance&per_page=5&media=photos&format=json&nojsoncallback=1"
        flickr_resp = requests.get(url=flickr_url, headers=headers)
        data = flickr_resp.json()["photos"]["photo"][0]

        image_url = 'https://farm'+str(data["farm"])+'.static.flickr.com/'+str(data["server"])+'/'+str(data["id"])+'_'+str(data["secret"])+'_n.jpg'

        self.hassapi.set_state("sensor.birdpic", state='on',
                               attributes={"image": image_url})

Importing MQTT Sensors into HomeAssistant

Now that we have all the sensors defined and communicating via MQTT, we have one more step to import them into HomeAssistant. This MQTT documentation by HomeAssistant is good to read about if you need a broker setup. I will not be going over the broker in this tutorial, but may add one in the future. I tend to like the yaml configuration for HomeAssistant, so for the sake of this guide, I’ll be referencing the manual configuration of MQTT items and sensors.

To add the sensors from above, open up your configuration.yaml file in your favorite editor. You’ll then want to add the mqtt platform and domain:

mqtt:
  - { domain }:

For the BirdNet sensors, we will be using a single domain: sensor. Feel free to copy and paste my config from below, but make sure the names of each entity align with your needs, syntax, and nomenclature/system.

Full MQTT Sensors in Configuration.yml

mqtt:
  sensor:
    - name: "Bird Common Name"
      state_topic: "birdnet/sensors/common_name"
    - name: "Bird Science Name"
      state_topic: "birdnet/sensors/science_name"
    - name: "Bird Time Seen"
      state_topic: "birdnet/sensors/time_seen"
    - name: "Bird Date Seen"
      state_topic: "birdnet/sensors/date_seen"
    - name: "Bird Confidence"
      state_topic: "birdnet/sensors/confidence"
      value_template: '{{ (value|float(0) *100) | round(1) }}'
      unit_of_measurement: '%'

You might be looking at the list above and wondering where the Flickr and Wikipedia Description entities are. They were already created by the AppDaemon script! Specifically, self.hassapi.set_state() function will either update the state for an exisiting entity or, if the entity doesn’t exist, it will create a new one.

For the rest of the mqtt payloads, we need HomeAssistant to create them as they come in, which is why we add the above code block to our HomeAssistant configuration file. To be clear, you do not need to add the Wikipedia and Flickr sensors to HA’s configuration file!

Adding the Camera entity

Last but not least, we need to add a camera entity to ensure that the sensor.birdpic can actually be rendered visually. It’s really easy to add this sensor, so this should be quick. Here’s how:

1. In HomeAssistant, navigate to Settings > Devices & Services > Integrations
2. Click “+ Add Integration” in the bottom right-hand corner. Alternatively, if you already have a camera integration enabled, look for the “Generic Camera” card and click “Add Entry”
3. Name the sensor. In this case, I called it “BirdPicturesfromFlickr” and renamed the entity to camera.birdnet_flickr.
4. You should now see a “Still Image URL” as the first of a few options on the screen. Enter the following into the still image field: {{ state_attr('sensor.birdpic', 'image') }} (This is the sensor we created in AppDaemon with the flickr url as the attribute).
5. Stream Source and RTSP transport protocol can both be left blank.
6. Authentication - select “digest”.
7. Username and Password can be left blank.
8. Frame Rate - 2
9. Leave the rest of the check boxes unchecked and click Submit

When you now click on the entity, you should see an image!

By this point, you should have successfully created 7 new sensors in HomeAssistant. In Part 2 of this article, we’ll take a look at Home Assistant, see what these sensors look like, and create a rudimentary dashboard.

Birdnet AppDaemon Script

import time
import requests


class birdnet(adbase.ADBase):
    def initialize(self):
        self.hassapi = self.get_plugin_api("HASS")
        self.adapi = self.get_ad_api()
        self.mqttapi = self.get_plugin_api("MQTT")
        self.birdnet_mqtt = "birdnet"
        self.mqttapi.listen_event(
            self.birdnet_message, "MQTT_MESSAGE", topic=self.birdnet_mqtt
        )

    def birdnet_message(self, event_name, data, kwargs):
        pre_split = data["payload"]
        common_name = pre_split.split(',')[0].strip()
        science_name = pre_split.split(',')[1].strip()
        date_seen = pre_split.split(',')[2].strip()
        time_seen = pre_split.split(',')[3].strip()
        week_seen = pre_split.split(',')[4].strip()
        confidence = pre_split.split(',')[5].strip()

        # print(f"A {common_name} was seen on {date_seen} at {time_seen}. Confidence is {confidence}.")

        self.mqttapi.mqtt_publish("birdnet/sensors/common_name", common_name)
        self.mqttapi.mqtt_publish("birdnet/sensors/science_name", science_name)
        self.mqttapi.mqtt_publish("birdnet/sensors/time_seen", time_seen)
        self.mqttapi.mqtt_publish("birdnet/sensors/date_seen", date_seen)
        self.mqttapi.mqtt_publish("birdnet/sensors/confidence", confidence)

        url = f"https://en.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&exintro&explaintext&redirects=1&titles={science_name}"
        response = requests.get(url)
        response = response.json()

        for value in response['query']['pages']:
            wiki_desc = response['query']['pages'][value]['extract']
            self.hassapi.set_state("sensor.birdnet_wiki", state='on',
                                   attributes = {"description": wiki_desc})

        headers = {'User-Agent': 'Python_Flickr/1.0'}
        flickr_api = "enter_your_api_key"
        flickr_url = f"https://www.flickr.com/services/rest/?method=flickr.photos.search&api_key={flickr_api}&text={common_name} bird&sort=relevance&per_page=5&media=photos&format=json&nojsoncallback=1"
        flickr_resp = requests.get(url=flickr_url, headers=headers)
        data = flickr_resp.json()["photos"]["photo"][0]

        image_url = 'https://farm'+str(data["farm"])+'.static.flickr.com/'+str(data["server"])+'/'+str(data["id"])+'_'+str(data["secret"])+'_n.jpg'

        self.hassapi.set_state("sensor.birdpic", state='on',
                               attributes={"image": image_url})

Do want to use both Github & and a Self-hosted Git Repo? Here’s how I’ve been doing it!

I really enjoy self-hosting services that I use everyday. One of those includes a git-style version control software. In my case, I’ve been running Gitea for a few years now and have been really satisfied with everything (except for that one time that an update broke all my templates).

At the same time, there’s the entire social element that comes with Github along with having your public repositories available in a place that other developers are already spending time on. Instead of adding, committing, commenting, and pushing on two different repos, here’s how I run all those commands just once and push it to both repos.

Note: An import git note to remember is that you can only push to multiple remote repositories. You’ll have to select which repo you want to be the main pull repository. Have this be remote-url-one in the below instructions.

Command Line Instructions

These instructions come after you initialize the repo in your directory. Make sure you have both of your remote git URLs handy at this point!

git remote add {{ remote-name }} {{ remote-url-one }}
git remote set-url --add --push {{ remote-name }} {{ remote-url-one }}
git remote set-url --add --push {{ remote-name }} {{ remote-url-two }}

To confirm that everything worked as expected, run git remote -v to check your remote repos. You should see one repo in there twice, once for (push) and once for (fetch).

I use the remote name “all” for multiple repos, so here’s what my git remote -v returns:

> git remote -v
all     https://git.rsmsn.co/Normanras/rsmsn_blog.git (fetch)
all     https://git.rsmsn.co/Normanras/rsmsn_blog.git (push)
all     https://github.com/Normanras/rsmsn_blog.git (push)
all     https://git.rsmsn.co/Normanras/rsmsn_ddblog.git (push)

To now push to your repositories, after adding and committing run git push {{ remote-name }} --all. My command is git push all --all (see why I use all, now?)

Here’s the man page description on the --all flag:

--all
    Push all branches (i.e. refs under refs/heads/); cannot be used
    Instead of naming each ref to push, specifies that all refs under
    end, locally updated refs will be force updated on the remote end,
    Do everything except actually send the updates.
    same as prefixing all refs with a colon.

And that’s it! You should be able to push everything to both of your repos fairly easily now with this new set commands.

For the last 3 days, I have been spending a few hours after working trying to figure out why my brand new Hugo site was not loading correctly on my sub-domain. For context, I use Nginx to host all my apps and servers, most of them using reverse proxy protocols such as $proxy_host, $forward_scheme, and $port. There are a few more and I’m happy to share some reverse proxy nginx config files. See my post on moving from NPM to Nginx for more information.

Once I got the basic idea of this blog up and running, I ran hugo and then used scp to send the files to my nginx host’s public folder. Despite index.html and all the CSS files being in the right spots, I kept getting a few repetitive errors from nginx - either in my browser’s console or in nginx’s log files. I swore I read through every StackOverflow or personal blog post I could find. In fact, the next day, I would sit back down after work to debug and I would continue to find the same sites and posts I found the day before! It was getting frustrating.

Wouldn’t you know… it was one of the simplest solutions that got it all working. Here’s a breakdown of what I was seeing and my hypothesis.

Errors

Console Errors:

• Incorrect MIME type –> css files being set as text/html type.
• 500/502 Errors when trying to load javascript files
• 500 Errors when trying to load child pages.

Nginx Log Errors for this server:

• [error] 1147432#1147432: *84013 invalid URL prefix in "://:/favicon-16x16.png"
• [warn] 1147432#1147432: *84013 using uninitialized "port" variable

Nginx error.log errors:

• [error] 1118832#1118832: *77105 directory index of "/var/www/html/" is forbidden

I thought I had tried everything, but it was a rip and replace from a single blog post that solved it for me. Some of the things I tried include the following. _Note: when I mention ’nginx config file’ below, I am referring to the specific file for this subdomain. I have a single global nginx.conf and then individual files for each subdomain/proxy host.

• Editing index.html to ensure that any referenced file had an explicit mime type associated with it.
• Included include { full path }/mime.types in the specific nginx config file.
• Included specific location ~ \.(css|js)$ { sections in my nginx config file.
• Tried assigning the $forward_scheme, $host, and $port variables (similar to a reverse proxy host).
• Removing any SSL references <– This caused similar behavior but different errors! I thought I was making progress…🫥
• Switched out the variables of root and alias between my server and location blocks.
• Started with something super simple, such as the suggestions from Gideon Wolfe’s Block.

If you clicked on the link I just shared, you’ll see that Gideon’s setup is quite similar to the one that I finally got to work. My thought there is that my errors were less about the nginx config setup from within the file, and instead it’s very possible that I set incorrect directory permissions after transferring all the public files to the web server. Gideon’s blog was the very first I clicked on, so I owe them my thanks since their site was the entry point to figuring all this out! You can find a list of all the blogs I stumbled upon in this weird and fluctuating journey of doing something as simple as sharing my static files on an nginx web server.

Solution

In the end, newbs.rocks blog post on setting up Hugo with nginx provided me a config example that worked for my setup. I think part of what happened was that as I was cycling through all the blog posts and StackOverflow posts, I would remove one or two lines (usually the one or two I changed from the previous post’s recommendations) but in doing that, was making more of a mess for myself, burying the error even more deeply. By replacing everything, I’ve brought it back to a manageable place.

You’ll also notice in my final config file, I was able to add back in the Authelia snippets, paths for the SSL certs, and a few other items that connect nginx to the rest of my infrastructure.

If you’ve stumbled upon this, I hope it helps you figure out your Hugo/Nginx issues! I definitely saw a lot of people posting in Hugo’s Discourse asking about mime type errors, so it is very likely that whatever you’re facing isn’t isolated to just you.

Now that I have this up and running, I need to write (and post!) a script that will pull from my repo, change directory ownership, and reload nginx.

Resources

Working Nginx Config File

# ------------------------------------------------------------
# selfhosted.rsmsn.co
# ------------------------------------------------------------

server {
    listen 80;
    listen [::]:80;

    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_name selfhosted.rsmsn.co;

    root /var/www/html/public/;
    index /index.html;

    # Force SSL
    include conf.d/include/force-ssl.conf;

    # Custom SSL
    ssl_certificate custom_ssl/npm-3/fullchain.pem;
    ssl_certificate_key custom_ssl/npm-3/privkey.pem;


    access_log /var/log/nginx/blog.log combined;
    error_log /var/log/nginx/blog_error.log warn;

    include snippets/authelia-location.conf;

    location / {
        try_files $uri $uri/ =404;
        include snippets/authelia-authrequest.conf;
    }
}

Blogs and Sites

• Gideon Wolfe - Deploying a static Hugo site with NGINX
• Pvera - Deploying a simple static website using Nginx and Hugo
• Hugo Support thread on Discourse
• BravosLab - Static website with Hugo and Nginx
• Cavelab - Hugo Build deploy to Nginx
• River - Serving Hugo from a non-root location with Nginx

A Tutorial Repo for migrating your Nginx Proxy Manager proxy setup to Nginx. I wrote this originally for this reddit post and to post this my Github profile. Thought my website would also be a good place to share it for any passers-by.

To give clear instructions to help users migrate from using Nginx Proxy Manager (NPM) to standard Nginx. This tutorial is not exhaustive and there are many other implementations of this transition. I would recommend checking out the many Nginx Documentation Sites and tutorials to learn more.

Introduction

If you’re anything like me and you got into the self-hosted/homelab/diy game sometime within the last 5 years, you’ve likely been recommended to use Nginx Proxy Manager as one of the choice Reverse Proxy services. If you’ve also been paying attention to various self-hosted communities, you may have also come across Christian Lempa’s Video on trusting smaller self hosted projects and tools.

Spoilers: He roasts NPM in his video and towards the end says he won’t be using NPM anymore. He also, perhaps purposely, doesn’t share which tool he will be migrating to.

Whether you follow Christian away from NPM or not, it dawned on me that while NPM is using a very trusted web server and reverse proxy under the hood, I hadn’t taken the time to understand how an Nginx Config actually worked. Since NPM was already creating most of the files for Nginx, I got to reading through all the files and reworking them so that I could begin using Nginx without the NPM gui.

Contributing: This is not all encompassing of Nginx possibilities. Including instructions for various installation methods, using OpenResty, and any other migrations or use cases would help the community. If you’d like to add in additional information on how to migrate from NPM to Nginx, that is welcome. Simply submit a PR with your steps.

TL;DR - Quick Steps

1. Copy the following contents (including sub-directories) from the NPM /data/nginx directory to the Nginx /etc/nginx folder:

   • proxy_hosts > sites-available
   • conf.d > conf.d
   • snippets > snippets
   • custom_ssl > custom_ssl (if applicable)

2. Edit each file in your sites-available directory and update the paths. Most will change from /data/nginx/ to /etc/nginx.

3. Edit your nginx.conf file and ensure the following two paths are there:

   • include /etc/nginx/conf.d/*.conf; and include /etc/nginx/sites-enabled/*;

4. Symlink the proxy host files in sites-available to sites-enabled

   • ln -s * ./sites-enabled

5. Test your changes with nginx -t. Make appropriate changes if there are error messages.

Pre-requisites & Assumptions

I am using an Ubuntu VM with NPM and it’s db as a Docker Container while Nginx is installed natively on the machine. You don’t have to use this setup exactly, but I am making a few assumptions as to what you should have access to before you begin. I am also using custom SSL certs, but theoretically, the transition should be the same when using Lets Encrypt.

I’ve added some example files to show before and after changes to this repo and outlined file trees below.

• You understand the basics of what a Reverse Proxy is doing and are sticking with some stock settings (like exposing port 80 an 443).
• You’ve installed NPM and Nginx using your preferred method.
• You have access to both NPMs file tree and Nginx’s.
• If using NPM in docker, make sure you’ve mapped a local volume on the host to the container.
• My setup using docker-compose is the following: /user/nginx/data:/data.
• Know where your Nginx files are. If using docker, same as above, make sure your container directories are mapped to the host.
• For a linux install, they should be accessible at /etc/nginx.
• You know how to edit files at the command line using nano, vi, vim, neovim, emacs, or something else.

Nginx Files

Nginx uses the nginx.conf file and within that file, it will include your proxy files. These exist under ./nginx/sites-enabled/. In the main nginx.conf file, the line include /etc/nginx/sites-enabled/*; will bring in those files to the config file, making the proxies accessible.

How to Transition - Detailed Version

1. Since NPM uses Nginx under the hood, they are both, by default, going to try and use ports 80 and 443 to serve up your apps and content. Turn off both systems.

   • Docker: docker stop [app_container, db_container]
   • Systemd: systemctl stop nginx

2. Copy your proxy_host (NPM) files to the sites-available (Nginx) folder. cp -r /user/nginx/data/nginx/proxy_hosts/* /etc/nginx/sites-available/

3. Nginx doesn’t really care what the files are called, but NPM numbers them based on the order in which you added them in the GUI. I find it better to rename them to what service they actually serve up for easier identification later.

4. Copy your custom_ssl folder from NPM to the custom_ssl folder in Nginx. See the following step for the various default paths in both systems.

   • cp -r /user/nginx/data/custom_ssl/* /etc/nginx/custom_ssl/

5. Copy the conf.d folder from NPM to the conf.d folder in Nginx. Note: For some reason, not all of the files in the proxy files were actually in my conf.d directory. If you’re missing any files, please download and/or copy and paste them from the NPM Repo

   • cp -r /user/nginx/data/nginx/conf.d /etc/nginx/

6. If you had any additional files included in the Advanced section of an NPM Proxy Host, make sure you copy them over. For my setup and this tutorial, they were all located in the snippets directory.

   • cp -r /user/nginx/data/nginx/snippets/* /etc/nginx/snippets/

7. There are a number of lines that need to be updated in each proxy configuration file to make them work with Nginx. I’ve placed additional comments in ./proxy_host/npm_proxy.conf file. The line changes are the following:

   1. Custom SSL path:

      • NPM path: /data/custom_ssl...
      • Nginx path: /etc/nginx/custom_ssl...

   2. conf.d:

      • The paths should remain the same. However, if you changed the path for conf.d in Nginx and differed in step 5, above, make sure you use the correct path.

   3. Access & Error Logs

      • NPM path: /data/logs/...
      • Nginx path: /var/log/nginx/...

8. Double Check all your paths! If this is your first time using Nginx, make sure every directory is correct! Save your work.

9. Navigate to the nginx.conf file which is located at /etc/nginx/. You can see the one I am using in this repo.

10. Make sure that you have the following two lines in the main conf file and that they are pointing to the appropriate directories in the nginx directory path.

    • include /etc/nginx/conf.d/*.conf; and include /etc/nginx/sites-enabled/*;

11. You’ll notice that you ensured there was a sites-enabled directory in the configuration file, but you changed all your proxy host config files in sites-available! Good eye, all that’s left is to symlink the files to sites-enabled so that nginx can start using them.

12. To symlink the available proxy files run the following command within the sites-available directory:

    • ln -s * ./sites-enabled

13. Once you’re confident that you’ve done all the above correctly, you can test your setup using nginx command and flags. While in the directory with your nginx.conf file - usually /etc/nginx - run the following command: nginx -t.

14. If all is working as expected you should see the below output. If it returns any errors, fix them appropriately. It will usually tell you what line is throwing the error. In this case, there’s a high likelihood that it will be path error.

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

And that’s it! You can now restart your nginx service on the host and access all your sites just as if you were using Nginx Proxy Manager! Make sure you take a look at your logs and system’s status should nginx fail to start.

Additional Information/Appendix

File Trees for NPM (in container) and Nginx (on host)

I did not expand every directory in these trees. Only the ones that are pertinent for reference in this tutorial.

NGINX

├── conf.d
│   └── include
│       ├── assets.conf
│       ├── block-exploits.conf
│       ├── force-ssl.conf
│       ├── ip_ranges.conf
│       ├── proxy.conf
│       └── resolvers.conf
├── custom_ssl
│   ├── npm-1
│   │   ├── fullchain.pem
│   │   └── privkey.pem
│   ├── npm-2
│   │   ├── fullchain.pem
│   │   └── privkey.pem
│   └── npm-3
│       ├── fullchain.pem
│       └── privkey.pem
├── fastcgi.conf
├── fastcgi_params
├── koi-utf
├── koi-win
├── mime.types
├── modules-available
├── modules-enabled
├── nginx.conf
├── proxy_params
├── scgi_params
├── sites-available
│   ├── auth.conf
│   ├── bitwarden.conf
│   ├── codehub.conf
│   ├── default.backup
│   ├── files.conf
│   ├── notes.conf
│   ├── photos.conf
│   ├── rsmsn-root.conf
│   ├── wordle.conf
│   └── wordle-it.conf
├── sites-enabled
│   ├── auth.conf -> /etc/nginx/sites-available/auth.conf
│   ├── bitwarden.conf -> /etc/nginx/sites-available/bitwarden.conf
│   ├── codehub.conf -> /etc/nginx/sites-available/codehub.conf
│   ├── files.conf -> /etc/nginx/sites-available/files.conf
│   ├── notes.conf -> /etc/nginx/sites-available/notes.conf
│   ├── photos.conf -> /etc/nginx/sites-available/photos.conf
│   ├── wordle.conf -> /etc/nginx/sites-available/wordle.conf
│   └── wordle-it.conf -> /etc/nginx/sites-available/wordle-it.conf
├── snippets
│   ├── authelia-authrequest-basic.conf
│   ├── authelia-authrequest.conf
│   ├── authelia-authrequest-detect.conf
│   ├── authelia-location-basic.conf
│   ├── authelia-location.conf
│   ├── authelia-location-detect.conf
│   ├── fastcgi-php.conf
│   ├── proxy.conf
│   └── snakeoil.conf
├── uwsgi_params
└── win-utf

NPM

├── data
│   ├── access
│   ├── custom_ssl
│   │   ├── npm-1
│   │   │   ├── fullchain.pem
│   │   │   └── privkey.pem
│   │   ├── npm-2
│   │   │   ├── fullchain.pem
│   │   │   └── privkey.pem
│   │   └── npm-3
│   │       ├── fullchain.pem
│   │       └── privkey.pem
│   ├── keys.json
│   ├── letsencrypt-acme-challenge
│   ├── logs
│   ├── mysql
│   └── nginx
│       ├── custom
│       ├── dead_host
│       ├── default_host
│       ├── default_www
│       ├── dummycert.pem
│       ├── dummykey.pem
│       ├── proxy_host
│       │   ├── 10.conf
│       │   ├── 11.conf
│       │   ├── 12.conf
│       │   ├── 13.conf
│       │   ├── 15.conf
│       │   ├── 1.conf
│       │   ├── 2.conf
│       │   ├── 4.conf
│       │   ├── 5.conf
│       │   └── 6.conf
│       ├── redirection_host
│       ├── snippets
│       │   ├── authelia-authrequest-basic.conf
│       │   ├── authelia-authrequest.conf
│       │   ├── authelia-authrequest-detect.conf
│       │   ├── authelia-location-basic.conf
│       │   ├── authelia-location.conf
│       │   ├── authelia-location-detect.conf
│       │   └── proxy.conf
│       ├── stream
│       └── temp
├── docker-compose.yml
└── letsencrypt
└── renewal-hooks

The project and library I was using was called Whiptail which allows you to use the terminal message boxes through a python script. Here are their docs. These message boxes are the same that you might see when you install a linux distro from a USB or first install of a server.

What I was using this for is to develop an visually appealing way to use the Meshtastic CLI. There can be a ton of settings and flags to add to your Meshtastic device and adding the flags one by one - or worse, you have to go back and change a flag and you’re not using zsh-vi-mode - can be time consuming. This project would allow you to choose in a whiptail dialog from a list of flags and pass values to the flags. By having them in a navigable list, you can always retract a flag you no longer need.

Then, once you’ve added everything you need, you confirm, and the command runs and syncs up your Meshtastic device!

Maybe I’ll post more about that project if I ever get around to finishing it. But since Meshtastic releases new updates on such a regular basis, I need to ensure the project pulls from Meshtastic commands in a more dynamic way.

Anyway, in this letter to no one, just thought I’d share my excitement.