<content:encoded><![CDATA[<p>Finding this little set of commands is one of the main reasons why I love the Command Line. Once I realized that almost everything that I was doing in a UI was possible on the CLI, the world opened up. One of the things I return to from time to time is when I can’t find a GIF I want to use in a message or email. Every time, I look up how to convert a video to a GIF and every time, I’m given a new solution. Imgur’s <ahref="https://imgur.com/vidgif">video-to-gif</a> used to be reliable, but the last few times it hasn’t been working as expected.</p>
<p>This last time, I was trying to convert a very important It’s Always Sunny in Philadelphia clip to a reaction gif. Can you believe there is no “thank you” reaction gif from when <ahref="https://www.youtube.com/watch?v=ROCKGuuviis">Dennis reads Charlie’s speech?!</a>. Here’s the gif for your own collections.</p>
<p><imgloading="lazy"src="../posts/img/thankyou.gif"alt="Charlie &amp; Dennis Thank you Gif"/>
</p>
<p>The command line functions I found came to get this done came mostly from <ahref="https://www.funkycloudmedina.com/2022/06/convert-a-video-file-to-a-gif-using-a-macos-automator-task/">Funky Cloud Medina’s</a> post on this. I didn’t want to use automator, but just write out a few commands, so here is what I did (on MacOS):</p>
<p>First, install ffmpeg and gifsicle with Homebrew. <code>brew install ffmpeg gifsicle</code></p>
<p>Next, navigate to the directory where your video is. If you plan on doing this regularly, you can create some permanent directories, but I started with creating two temporary directories for the images and then the final gifs.</p>
</code></pre><p><code>Untitled.mov</code> is the name of the video file in the directory you’re currently in and outputs each frame to the <code>pngs</code> folder with increment digits. The incrementing digits are so you don’t overwrite everything and end up with just a single picture file.</p>
<p>Next, we’ll use <code>sips</code>, the Scriptable Image Processing System. (Check out <code>man sips</code> from your own CLI for more info!)</p>
<pretabindex="0"><code>sips -s format gif pngs/*.png --out gifs
</code></pre><p>Almost there! This is processing all the image files into the gifs folder. Let’s now move into the gifs folder with cd: <code>cd gifs</code></p>
<p>And now we can use <ahref="https://www.lcdf.org/gifsicle/">gifsicle</a> to merge all the images into a single gif! We’ll do that with the following command:</p>
<title>How to Revisit your Terminal Session's History</title>
<link>/posts/save_terminal_to_file.html</link>
<pubDate>Tue, 19 Mar 2024 11:00:53 -0400</pubDate>
<guid>/posts/save_terminal_to_file.html</guid>
<description>Go beyond zsh/bash_history and save both your commands and the output to a file for later review! This command will help any homelabber that struggles with documentation for their network and services.</description>
<content:encoded><![CDATA[<p>I can’t believe I didn’t know about this command beforehand. When I first got into self-hosting and homelabbing, one of the app ideas I had that would help me with internal documentation is a terminal program that saves your input and output to a file for you to review later and extract the key commands you used. I also had the bonus idea that you could add comments as you were writing out commands.</p>
<p>Little did I know at the time that comments from the CLI were already possible! I’ve already begun using comments which has been helpful if I need to look back at my <code>zsh_history</code> file. Here’s an example of a command I would use with my docker services.</p>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-bash"data-lang="bash"><spanstyle="display:flex;"><span>docker logs <spanstyle="color:#f92672">{</span>container_name<spanstyle="color:#f92672">}</span> --since 5m ;: The container is failing <spanstyle="color:#ae81ff">5</span> minutes after startup.
<p>You can write comments in <code>bash</code> and <code>powershell</code> as well!</p>
<ul>
<li>Powershell: <code>rem</code></li>
<li>ZSH: <code>;:</code></li>
<li>Bash: <code>#</code></li>
</ul>
</div>
<p>Turns out, I don’t need a fancy app to log the input/output of a terminal session! It exists in most unix based systems already and the command is simply this: <code>script</code>. That’s it!</p>
<p>Unlike <code>.zsh_history</code> or <code>.bash_history</code> which only saves the commands you input into the terminal, <code>script</code> will save both your commands <em>and</em> it’s output. Why didn’t I know about this before I started setting up my homelab?!</p>
<p>The next time I want to setup a new service or debug something on a server, I will now make sure I start my session with <code>script {date}-{service_name}.txt</code> and then start writing out commands. As long as I can remember to write my inline comments during the session, looking back and trying to figure out what I was thinking at the time should be a breeze! I can dump these files into my internal wiki as placeholders. Then, ideally, I’ll remove the commands that lead me down a dead end, clean up and expand comments, and easily keep my wiki growing. Ideally being the key word here… I’ll take a dump of history files for the time being.</p>
<p>Either way, I will literally have the output of my brain when doing CLI work in a file. Incredible.</p>
<p>Here’s a quick overview about script, taken from <code>man script</code>.</p>
</span></span><spanstyle="display:flex;"><span> The script utility makes a typescript of everything printed on your terminal. It is useful <spanstyle="color:#66d9ef">for</span> students who need a hardcopy record of an interactive session as proof of an assignment, as the typescript file can be
</span></span><spanstyle="display:flex;"><span> printed out later with lpr<spanstyle="color:#f92672">(</span>1<spanstyle="color:#f92672">)</span>.
</span></span><spanstyle="display:flex;"><span>
</span></span><spanstyle="display:flex;"><span> If the argument file is given, script saves all dialogue in file. If no file name is given, the typescript is saved in the file typescript.
</span></span><spanstyle="display:flex;"><span>
</span></span><spanstyle="display:flex;"><span> If the argument command is given, script will run the specified command with an optional argument vector instead of an interactive shell.
<description>Learn about running a quick automation that turns rows in your Google Sheets and plugs it into a Google Slide template to easily share more attractive content.</description>
<p>In case you haven’t heard of Google Apps Scripts, it is a built in Google IDE that uses Javascript to make various
Google products available via automation and scripting. While Google saves the files as <code>.gs</code> files, it’s just javascript,
don’t worry! You can learn more about it <ahref="https://www.google.com/script/start/">here</a> and <ahref="https://developers.google.com/apps-script/overview">here</a>.</p>
</div>
<p>Below, I’ll share the pieces of the script to explain what is going on. If you know what you’re doing and just want the
script, feel free to head to the bottom of the page to see the <ahref="#full-script">full script</a>.</p>
<h1id="goal">Goal</h1>
<p>The goal here was pretty straight forward. I wanted to add a button on the Google Sheet that allows the user to create a
Slides presentation from a subset of data within the sheet. The data will be date based, as this is what most end users need,
and the user will be able to pick from the currently available dates & populated data.</p>
<h1id="setup">Setup</h1>
<p>Getting setup, you need both a Sheet with a few headings and an empty Slide presentation with a few empty text boxes. The
empty text boxes will be key to helping us connect the data in the sheet to the correct placement in the slide. Here are two
screenshots of what this looks like for my example:</p>
<p>Ignoring any design from my screenshots - all credit goes to my much more creative wife - the setup for the sheet is fairly
simple. You need various headings in Row A of the sheet which we will be using to reference data. In this tutorial, our
headings are <code>Date, Topic 1, Topic 2, Topic 3, Additional Notes</code>. Whether you start on Row 1 or after that doesn’t matter too
much.</p>
<p>For the date column, we’ll be formatting our date like this: “February 26, 2024”. You’ll see why in a little bit.</p>
<p>On top of the actual data in the Sheet, the Apps Script is going to live in this document and just push data to the Slide. To
access your App Scripts, click Extensions > Apps Scripts. A new tab will open with a blank IDE style interface and an empty
<code>myFunction</code>.</p>
<h2id="slides-setup">Slides Setup</h2>
<p>Don’t worry too much about the design for Slides, you can change that later. But the important step is creating the empty
text boxes. After you create your text boxes (4 will be used in this tutorial), right click one of them and select “Format
Options”. A panel on the left hand-side should slide out. Click the “Alt Text” drop down, and then “Advanced Options”. That
little text box is the title for your text box; it is not used in the visual representation of the box, we will just be using
it as a reference point.</p>
<p>For ease of this tutorial, make the Title of the text box the same as the Header row from when we set up the Sheet, above.
Once you’ve added title to each of the text boxes, let’s head into the code.</p>
<h2id="onopen-function">onOpen Function</h2>
<p>The first function you need for creating a UI change in the Google Sheet is an <code>onOpen</code> function that will setup the UI when
</div><p>This first section we’re just getting the sheet ready for analysis. With the <code>dates</code> variable, we’re just looking at the
second column. The reason we’re using the <code>getDisplayValues()</code> method is because Google will automatically convert the dates
to include time zone, time, etc. We want to keep the date in the same format for a better user experience.</p>
<p>For that last line, we’re getting the current date & time in epoch time so we can run a comparison further down the script.
For this use case, we don’t need to include any dates in the past.</p>
</div><p>In this section we’re creating an empty array and instantiating a RegExp to ensure we have an actual date in the cell. See
<ahref="https://regex101.com">Regex101</a> to learn more about Regex and test different regex syntax.</p>
<divclass="box box-shortcode tip">
<spanclass="icon-box baseline">
<svg><usehref="#tip-box"></use></svg>
</span>
<p>Something that threw me off when first writing this regex function was the way the Google IDE manages escape character and
slashes. If you take the second line above and input it into <ahref="https://regex101.com">Regex101</a> you’ll see the <code>\s</code> or <code>\d</code>
become dark gray, basically skipping over that token. However, for Google, you’ll need an additional backslash to escape and
make the token become used by the function.</p>
<p>Here’s the “correct” RegExp string for Regex101.com: <code>^[A-Za-z]{3,15}\s\d{1,2},\s\d{2,4}</code></p>
</div>
<p>Next, we dive into a for loop, looping through the <code>dates</code> column of values (which we just called earlier). After attributing
each value to the <code>var date</code> variable, we also convert that same value into epoch time (by creating a new <code>Date().getTime()</code>)
so that we can compare it with today’s date.</p>
<p>After the variables are setup we need to check that the date isn’t empty; we don’t need any rows where a date hasn’t been assigned to it
yet. If we have a non-empty date value, let’s compare it using the regex string. All we’re doing here is asking “Is this date
in the format I’m expecting it?” If true, let’s keep the value and continue using it. If not, just ignore it.</p>
<p>So we’ve now found a value that’s in the date format we expect, let’s now take that same value in epoch time (referenced by
the <code>epochSheetDate</code> variable) and compare it to today’s epoch time date. If today’s date is less than the value in the
sheet, that means the date in the sheet is in the future.</p>
<p>So now we have a date in the correct format and that is at some future date from today. Fantastic! Once we’ve gone through
those checks, we’re ready to add the date to the array we created at the top of this section. Push on!</p>
</span></span><spanstyle="display:flex;"><span><spanstyle="color:#66d9ef">var</span><spanstyle="color:#a6e22e">response</span><spanstyle="color:#f92672">=</span><spanstyle="color:#a6e22e">ui</span>.<spanstyle="color:#a6e22e">prompt</span>(<spanstyle="color:#e6db74">'Which date would you like to create a presentation for? Please copy and paste the date exactly as you see it in the options below. Options: \n \n'</span><spanstyle="color:#f92672">+</span><spanstyle="color:#a6e22e">formattedDates</span>);
</span></span><spanstyle="display:flex;"><span><spanstyle="color:#a6e22e">ui</span>.<spanstyle="color:#a6e22e">alert</span>(<spanstyle="color:#e6db74">"Woohoo! Let's make a presentation!"</span>)
</span></span><spanstyle="display:flex;"><span><spanstyle="color:#a6e22e">ui</span>.<spanstyle="color:#a6e22e">alert</span>(<spanstyle="color:#e6db74">"Ouch. Looks like you entered something incorrectly (i.e. you entered nothing). Try again."</span>)
</span></span><spanstyle="display:flex;"><span><spanstyle="color:#66d9ef">var</span><spanstyle="color:#a6e22e">response</span><spanstyle="color:#f92672">=</span><spanstyle="color:#a6e22e">ui</span>.<spanstyle="color:#a6e22e">prompt</span>(<spanstyle="color:#e6db74">'Which date would you like to create a presentation for? Please copy and paste the date exactly as you see it in the options below. Options: \n \n'</span><spanstyle="color:#f92672">+</span><spanstyle="color:#a6e22e">stringlist</span>);
</span></span><spanstyle="display:flex;"><span><spanstyle="color:#a6e22e">ui</span>.<spanstyle="color:#a6e22e">alert</span>(<spanstyle="color:#e6db74">"Woohoo! Let's make a presentation!"</span>)
</span></span><spanstyle="display:flex;"><span><spanstyle="color:#a6e22e">ui</span>.<spanstyle="color:#a6e22e">alert</span>(<spanstyle="color:#e6db74">"Ouch. Looks like you entered something incorrectly (i.e. you entered nothing). Try again."</span>)
<description>Every time my OpenResty package tries to update, I can&#39;t restart the service with weird errors that don&#39;t point exactly to the problem. Here&#39;s how I&#39;ve learned to fix it.</description>
<content:encoded><![CDATA[<p>Turns out I wrote this back in November, but on a different machine and I never pushed my changes up to git. So here it is
now!</p>
<p>The last two times I’ve run <code>apt update && apt upgrade -y</code> on my web server and <ahref="https://openresty.org/">OpenResty</a> has an update to push out, I will
get some errors including Openresty is not setup correctly. There are a few other errors in the <code>systemctl status openresty</code>
output such as <code>process 223478 [nginx] is still running</code>. I might be slightly paraphrasing the errors, but that’s
roughly what I’m finding.</p>
<p>Like any debug session, I make sure nginx is disabling and not running (<code>systemctl disable</code> and <code>systemctl stop</code>), which I
can confirm. Now, Openresty does use Nginx under the hood, so that errors makes me think it’s just conflicting services
trying to run on top of each other.</p>
<p>The weirder part is when it warns me that OpenResty is not setup correctly. I didn’t change anything… so what is the
update/upgrade trying to setup?</p>
<p>Next, I’ll look through my config files (i.e. <code>nginx.conf</code>) - no changes there either and nothing that stands out as out of
the ordinary.</p>
<p>One of my bad habits is that when doing this sort of debugging and running something like <code>journalctl --since 21:45:00</code>, I’ll
look at the logs more closely with each try and fail. I should just look more closely from the beginning! But I digress.</p>
<p>Since the nginx process and openresty setup errors are the most plentiful but yield nothing, I’ll look back through the logs
for the single lines that I miss on my first few passes. That’s when I see it, buried between the other errors. A single line
that says my pid file can’t be found.</p>
<p>My pid file for OpenResty and Nginx is stored in my <code>/run</code> directory, but for whatever reason, whenever OpenResty pushes out
an update, it overwrites my systemd file and starts looking for the pid file under <code>/usr/local/openresty/nginx/logs/nginx.pid</code>.
Why the file would be located under a logs directory is still beyond me, but a quick update to the systemd file, a
<code>systemctl daemon-reload</code> and then <code>systemctl start openresty</code> and all my public facing services are back in action.</p>
<p>After I fixed it for the second time, I realized I hadn’t written this down in my documentation. So I’ve now recorded it and
thought I’d share it here in case it helps anyone else. I also did a bit of research after and found that even in this
<content:encoded><![CDATA[<p>This week, I’ve had to make some changes to an automation we had setup for a customer in <ahref="https://workato.com">Workato</a>. The
original recipe was made by a co-worker with a bunch of javascript nodes. Even though I’m customer facing and generally not
considered a technical employee, I knew the engineer who worked on this was swamped, so I decided to jump in.</p>
<p>In order to greatly reduce the number of nodes needed in the single recipe, I created a list of dictionaries in python (with
one of the values being another list!) and a quick for loop to grab all the appropriate elements from the various
dictionaries. I had thankfully already copied all the JS nodes to a single file for easy reference and retrieval. That one
file had over 8,000 lines, and I needed to replace all the javascript elements with python.</p>
<p>Here’s one node of Javascript that I was working with:</p>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-javascript"data-lang="javascript"><spanstyle="display:flex;"><span><spanstyle="color:#75715e">// @param input fields supplied in the recipe step, as an object
</span></span></span><spanstyle="display:flex;"><span><spanstyle="color:#75715e">// @return object matching the output schema
</span></span></span><spanstyle="display:flex;"><span><spanstyle="color:#75715e">// Eg: Code for returning time zone for an IP address
</span></span></code></pre></div><p>Most nodes were very similar, all except that their <code>all_domains</code> array had anywhere from 1 to 500+ elements. Thanks to the
neovim plugin <ahref="https://github.com/jonarrien/telescope-cmdline.nvim">Telescope Cmdline</a> by <ahref="https://github.com/jonarrien">Jon Arrien</a>,
you can very easily see your past history of Neovim commands. I’ve gone back through and pulled out a few special ones that
helped and that I was very surprised how well they worked. I don’t claim that this was the fastest or best way to clean up a
large file with repeating functions, but I definitely learned a bunch about searching-and-replacing in Neovim, and any future
needs for this will be <em>way</em> faster! One thing that has been really helpful with Neovim is that as you run the first portion
of all the commands below, Neovim should highlight what it finds. This is really helpful for real-time debugging of your
regex.</p>
<h2id="commands-and-quick-explanations">Commands and Quick Explanations</h2>
<p><strong>Explanation:</strong> The g in the command stands for global (hint, run <code>:h :g</code> in vim!) and will search across the entire buffer
you have open. In this command I’m looking for all Javascript single-line comments. Since the forward slash is a special
character for neovim, you need to escape it with a backwards slash, hence the odd looking <code>\/</code>. The second forward slash is
the next parameter neovim is looking for, and by adding +{num}, you can search for the pattern plus a number of lines.
Closing out the command (again, next parameter comes after the forward slash) with <code>d</code> for delete will then delete everything
with you searched for, plus the additional number of lines.</p>
<p>Bonus: if you wanted to make sure that you only pulled lines that <em>began</em> with a comment, you’d add a carrot to the beginning
<p><strong>Explanation:</strong> Moving on from <code>g</code> to <code>s</code>, the <code>s</code> here stands for substitute. There’s one big difference in using <code>g</code> and
<code>s</code>, though. While <code>g</code> will search globally, a singular <code>s</code> will only search on your current line. To search the entire
file/buffer, you need to include the percent character before the <code>s</code>. The rest of this is fairly straight forward, I am
searching for the declaration of a javascript variable and then substituting it with a python dictionary key. Since both
javascript and python declare arrays with square brackets, I wanted to make sure I kept that after the substitution. Lastly,
unless you only want the subsitute to change the first occurrence of what it finds, you’ll need to close out the command with
the <code>g</code> flag. Neovim’s docs say:</p>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-markdown"data-lang="markdown"><spanstyle="display:flex;"><span>[g] Replace all occurrences in the line. Without this argument,
</span></span><spanstyle="display:flex;"><span> replacement occurs only for the first occurrence in each line. If the
</span></span><spanstyle="display:flex;"><span>'gdefault' option is on, this flag is on by default and the [g]
</span></span><spanstyle="display:flex;"><span> argument switches it off.
<p><strong>Explanation:</strong> Really similar to the above command, but in this case, there are no characters I needed to save
post-substitution. We already know what <code>%s</code> does, so the pattern I’m looking for is all lines that start with
<code>exports.main</code>. Since I don’t need anything after that start of the line, I included <code>.*</code> which looks for any character until
the end of the line. After the pattern I write out what I need to replace it with, which is the first portion of the
dictionaries I need. Finally, we close it out with a global change so it changes it everywhere.</p>
<p><strong>Explanation:</strong> I won’t go over all the characters and regex commands as I’m reusing a lot of them from the above commands.
What is different here is that I’ve lumped the wild card in parenthesis like so: <code>(.*\)</code>. Now, I can call back what is
pulled by using <code>\1</code>. The command above is looking for a dictionary item with a list as the value. I didn’t need the list,
just a string, since all values in my list of dictionaries are single values. By searching for <code>"props": [ string ],</code> and
using the <code>\1</code>, I’m able to keep the string intact and the result is <code>"props": string,</code>.</p>
<p>This was the milestone command to learn for me and helped unlock so much potential in my brain for other use
cases. For all the other commands I’ve gone over above, I’m fairly simply just replacing characters I no longer need that
exist either <em>before or after</em> characters I do need. Now that I can keep strings intact for substitutions, I bet I could
revisit all the above commands and make them even more efficient.</p>
<p>What I’d like to learn next is if I can use an “or” statement in the regex. Let’s say I have the same situation as above - a
list of dictionaries with some dict values being a list. Well, if the lists are all formatted differently, some on new lines,
others on a single line, can I select both in a single command? Something like: <code>:%s/"domains": \[\(.* OR \n\)/</code>.</p>
<p>That will be a post for another time! Let me know on Mastodon if you have any other Neovim search and replace tips and tricks
that have become invaluable to your workflow. I’m always amazed and impressed with this community and how differently people
are able to use the same tool.</p>
<hr>
<p>By the way, here’s the final output of running the above commands on the sample code above, converted to Python.</p>
<p>Here are some various links of resources that I found helpful when learning about the above commands. I’m assuming you’ve
already read through anything in <ahref="https://neovim.io/doc/user/">Neovim’s Helpdocs</a> (<code>:h</code>), but sometimes you need a different take and explanation.</p>
<ul>
<li><ahref="https://www.reddit.com/r/neovim/comments/i5iptq/vim_search_find_and_replace_a_detailed_guide/">Vim Search, Find & Replace - Reddit</a></li>
<li><ahref="https://vim.fandom.com/wiki/Search_across_multiple_lines">Search Across Multiple Lines - Vim Tips Wiki</a></li>
<li><ahref="https://thevaluable.dev/vim-search-find-replace/">Vim Search and Replace with Examples - The Valuable Dev</a></li>
<li><ahref="https://stackoverflow.com/questions/62751398/how-to-remove-specific-characters-in-vi-or-vim-editor">Removing Specific Characters in Vim - Stack Overflow</a></li>
<pubDate>Fri, 20 Oct 2023 18:38:13 -0400</pubDate>
<guid>/posts/mini_neovim.html</guid>
<description>Neovim is already super efficient and lightweight, but sometimes I need a barebones config for my servers and remote, miniature development environments.</description>
<content:encoded><![CDATA[<p>If I’m being honest, when I started this quick project to reduce my Neovim plugin footprint size, I thought I was going to be
able to get rid of <em>way</em> more plugins! Inspired by <ahref="https://www.reddit.com/r/neovim/comments/179zawc/my_new_config_based_entirely_on_mininvim/">this Reddit
post</a> that setup a Neovim config
using only <ahref="https://github.com/echasnovski">echasnovski’s</a><ahref="https://github.com/echasnovski/mini.nvim">mini</a> library, I was
reminded that some of my remote server environments were still using Packer as their plugin manager, where my main machine
has upgraded to Lazy. In addition, I have since expanded my main config with more applicable plugins and key maps and the
Packer config was in a single neovim repo, as opposed to my more recently implemented <ahref="https://github.com/Normanras/dotfiles">dotfiles</a> setup.</p>
<p>Right now, I am using a whopping 77 plugins. That’s mostly because I’m not great at cleaning out old plugins I don’t use as
much anymore. Since I am client facing, a lot of them are around making my Markdown notes easier to manage, and Python plugins
to help with scripting and file management in the terminal. This poll shows 10-25 plugins being the most common amount.</p>
<blockquoteclass="reddit-embed-bq"style="height:240px"data-embed-height="240"><ahref="https://www.reddit.com/r/neovim/comments/14jdkbp/how_many_plugins_in_your_config_list_your/">How many plugins in your config. List your favorites!</a><br> by<ahref="https://www.reddit.com/user/Bamseg/">u/Bamseg</a> in<ahref="https://www.reddit.com/r/neovim/">neovim</a></blockquote><scriptasync=""src="https://embed.reddit.com/widgets.js"charset="UTF-8"></script>
<p></p>
<p>Once I got my dotfiles to my server environment, I got to work and got rid of all the superfluous plugins like additional
color schemes, todo-trouble, or others that I considered nice-to-have. After all, across all my VMs I’m mostly editing yaml,
json, and python scripts and I’m not in there for a very long time. If I was a real pro, I would just use out-of-the-box Vim,
but I really like the speed some key maps and plugins have given me.</p>
<p>You can check out my git branch <ahref="https://github.com/Normanras/dotfiles/tree/neovim-only-minimal">here</a>. If you decide to clone it, make sure you symlink <code>.config</code> folder from a <code>~/.dotfiles/</code> directory to <code>~/.config/</code>. I’ve been using
<ahref="https://www.gnu.org/software/stow/manual/stow.html">stow</a>, but any symlink tool will work.</p>
<p>If you have a neovim config that you’d like to “minimize”, see some steps below. For any dotfile config - Neovim, tmux, zsh -
I like to make these changes in a clean environment. I want to replicate as closely as possible what a fresh install and
<li>And that’s it! Go check your Github/Gitlab and check your branches and commits. You should see everything there! Congrats.</li>
</ol>
<p>For that last command, you can always check what your remote repo’s push name is by running <code>git remote -v</code>. Most often, the name will be something like <code>origin</code> or <code>main</code>. If you saw <ahref="/posts/multiple_git_remotes.html">my post on having multiple remote repos</a>, you may something like “all”. Just make sure to check before pushing!</p>
<content:encoded><![CDATA[<p>This is a quick treat! We recently learned that one of our children needs to take medicine twice a day for the foreseeable
future. He’s too young to take it on his own, so the twice-a-day responsibility is split up between my partner and
I. However, sometimes our schedules don’t overlap so succinctly, so we needed some sort of indicator to let the other know if
the previous dose was (or wasn’t!) taken. Naturally, I sprung into action with an ESP device and components.</p>
<h2id="device-overview">Device Overview</h2>
<p>The device works like this (<ahref="/posts/medicine_indicator_light.html#pictures">see the pictures below</a>): Two <ahref="https://www.adafruit.com/product/1260">single neopixel LEDs</a> and a
non-latching button are connected to the ESP8266. Inside the code there is a <ahref="https://www.arduino.cc/reference/tr/language/structure/control-structure/switchcase/">switch-case</a>
section which changes states per high/low state of the button. On each press, the lights will change to one of the following:</p>
<ul>
<li>Red/Red</li>
<li>Green/Red</li>
<li>Red/Green</li>
<li>Green/Green</li>
</ul>
<p>For each case, I established a values for <code>morning</code> and <code>evening</code> which will be given a value of 0 or 1 - 0 to indicate that the medicine was
not taken, and 1 to indicate that it was taken. Along with the time server data, I send an MQTT payload to my <ahref="https://mosquitto.org/">MQTT Broker</a>.</p>
</span></span></code></pre></div><h2id="thoughts-and-background">Thoughts and Background</h2>
<p>I’d like to write a longer post that explains each section of the code, but I’m running out of
time - work has been quite busy lately. If you’d like me to add a post with more code explanations, or just have questions,
feel free to mention me on <ahref="https://fosstodon.org/@notnorm">Mastoton/Fosstodon</a> and we can chat.</p>
<p>The much shorter explanation is that I didn’t want this device to be an isolated device in my network; this was part of the
impetus in using an ESP device as opposed to another microcontroller. So what I did was connected the ESP8266 to my wifi
network and imported an NTP (Network Time Protocol) server to ensure I get I have the current time. Then, look at the state
given by the case (and thus the external light indicator), and construct and transmit an MQTT payload with the state of the
light & time, so that I can use it elsewhere. In other words, now with MQTT payloads being accessible to HomeAssistant, I can
send myself a notification, or make an announcement on a speaker.</p>
<p>What’s neat about importing a time server is that at midnight, I reset both of the lights to red so that I don’t have to
reset it manually when I get up in the morning.</p>
<p>In a separate post, I’ll show how I use the MQTT payload in HomeAssistant to send <ahref="https://pushover.net">Pushover Notifications</a>
to me and my partner in case we forgot to give my child the medicine! What’s great about this little project is that it is
scalable (add more components and sensors!) and it isn’t limited to a medicine indicator. You
could use the button to count how many times your dog (or child!) has gone to the bathroom, how many times you have eat or
drink water while working, and many more ideas.</p>
<p>Assuming everything has gone well for the day and we’ve given my child both doses, we should be heading to bed with both of
those lights being green.</p>
<h2id="materials-used">Materials Used</h2>
<p>Here is what I used to construct this:</p>
<ul>
<li><ahref="https://www.amazon.com/HiLetgo-Internet-Development-Wireless-Micropython/dp/B081CSJV2V/ref=sr_1_1?crid=7JMC6TOMCS9I&keywords=hiletgo+esp8255&qid=1697229200&sprefix=hiletgo+esp8255%2Caps%2C89&sr=8-1">ESP8266 Dev
Board</a> - I tend to like HiLetgo from Amazon.</li>
</span></span><spanstyle="display:flex;"><span><spanstyle="color:#75715e">#define NEO_PIN 14 </span><spanstyle="color:#75715e">// Pin for all the Neopixels
</span></span></span><spanstyle="display:flex;"><span><spanstyle="color:#75715e"></span><spanstyle="color:#75715e">#define SIG_PIX 2 </span><spanstyle="color:#75715e">// Data pin for Neopixels
</span></span><spanstyle="display:flex;"><span><spanstyle="color:#66d9ef">int</span> mode <spanstyle="color:#f92672">=</span><spanstyle="color:#ae81ff">0</span>; <spanstyle="color:#75715e">// Active mode on startup
</span></span><spanstyle="display:flex;"><span><spanstyle="color:#66d9ef">float</span> value <spanstyle="color:#f92672">=</span><spanstyle="color:#ae81ff">0</span>;
</span></span><spanstyle="display:flex;"><span>
</span></span><spanstyle="display:flex;"><span><spanstyle="color:#75715e">// Setup defaults for the eventual MQTT payload
</span></span><spanstyle="display:flex;"><span><spanstyle="color:#75715e">// Print Status of Wifi Connection to Serial Monitor
</span></span></span><spanstyle="display:flex;"><span><spanstyle="color:#75715e"></span> Serial.println(<spanstyle="color:#e6db74">"Attempting to connect to SSID: "</span>);
<description>A Follow up from the previous post, this tutorial takes all the sensors we created and loads them into a beautiful dashboard!</description>
<content:encoded><![CDATA[<h2 id="checking-for-entities">Checking for Entities</h2>
<p>If you’re following up on this from <ahref="/posts/birdnet_homeassistant.html">my first post</a>, you’ve already added your AppDaemon script and confirmed that the AppDaemon logs don’t show any errors. Now is the true test if it’s working: do you have the
<p>The best way to do this is by just type <code>e</code> from any screen in the HomeAssistant UI! That will bring up a list of entities.
Start typing “bird” or “birdnet” and you should see the new entities listed there.</p>
</div>
<h2id="dashboard-overview--dependencies">Dashboard Overview & Dependencies</h2>
<p>Now that we have the correct entities, lets take a look at what we’re working with. Full disclosure that once I got this
working, I haven’t really revisited it, refactored it, or made any improvements. I’m sure you’ll find ways to use less YAML,
but I wanted to get this out there sooner than later!</p>
<p><imgloading="lazy"src="../posts/img/birdnet-homeassistant-dash-full.png"alt="HomeAssistant BirdNet-Pi Dashboard - Full View"/>
</p>
<p>I’ve included the code for all the cards at the bottom of this post. You can find them <ahref="/posts/birdnet_homeassistant_part2.html#dashboard-yaml">here</a>.
This dashboard is pretty simple, it brings in almost all of the sensors
we created in the first post and organizes them in an as-pleasant-as-possible view. I’m definitely not a designer, so some of
</span></span></code></pre></div><p>The thing is, HomeAssistant has already implemented this really neat feature for calculating time, especially from when
something was last updated. This function is called <code>relative_time</code>. Having something like this allows you set automations
to run after a specific amount of time has passed since the last time a sensor or entity was updated.</p>
<divclass="box box-shortcode tip">
<spanclass="icon-box baseline">
<svg><usehref="#tip-box"></use></svg>
</span>
<p>An idea! 💡 For our specific use case, you could set up an automation that sends you a notification if no birds have been detected for
over 30 minutes. Of course, we’ll set parameters like not to notify you at night or during the winter months.</p>
</div>
<p>The issue I faced with relative time has to do with the sensors I created from my AppDaemon script. Relative time expects
a date <em>and</em> time. I was only passing the time. In Home Assitant if you use the Developer Tools > Template to test relative
time out on the <code>sensor.bird_time_seen</code> sensor, you’ll get a result of 126 years… That’s because without a date, Home
Assistant defaults the date to <code>1900-01-01</code>. The full relative_time return is <code>1900-01-01 15:15:15</code>.</p>
<p>We could go back and set the sensors to include both date and time, but I prefer them separate so that I can use them in
different places. For this dashboard, the day is always today, so having the date felt redundant. To create a new sensor
using the <code>relative_time</code> function, you’ll need to edit your <code>configuration.yaml</code>.</p>
<p>Once you’re editing your config file, add the following:</p>
</span></span></code></pre></div><p>What this does is uses Home Assistant’s templating functionality and creates a new sensor called “Bird Last Seen”. The
default <code>sensor.</code> name will be <code>sensor.bird_last_seen</code>.</p>
<p>To configure the state of that sensor, we first set a variable called <code>birdseen</code>. To this variable we are assigning the
concatenated values of <code>bird_date_seen</code>, a single whitespace, and <code>bird_time_seen</code>. We’re choosing this format because that
is the format that <code>relative_time</code> returned before when we tried using it without a date.</p>
<p>As a quick experiment, take the templating code under the <code>state: ></code> parameter above and throw it into Developer Tools >
Template. Do you get 126 years? Or something more realistic? If something more realistic, amazing!</p>
<p>We’re almost there! Here’s what you should see in HomeAssistant if the sensor was created correctly.
<imgloading="lazy"src="../posts/img/birdnet-homeassistant-birdseen-sensor.png"alt="Bird Last Seen Entity"/>
</p>
<divclass="box box-shortcode info">
<spanclass="icon-box baseline">
<svg><usehref="#info-box"></use></svg>
</span>
<p>If you’re new to templating for Home Assistant (or in general!) it would be helpful to read through a few of the docs that
<content:encoded><![CDATA[<p>This is Part One of a Two Part Series. You can find Part Two, <ahref="/posts/birdnet_homeassistant_part2.html">here.</a></p>
<p><strong>Update: 10/11/2023. A huge thanks to Mastodon User <ahref="https://mastodon.social/@e_mobil2014">e_mobile2014</a> who found a broken link in this guide and pointed out that I never explained how to get the mqtt sensors into HomeAssistant!</strong></p>
<h2id="what-you-will-need">What you will need</h2>
[ ] Notify each new infrequent species detection (< 5 visits per week)
[ ] Notify each new species first detection of the day
[X] Notify each new detection
[X] Send weekly report
Minimum time between notifications of the same species (sec): 5
</code></pre><p>To test my MQTT notifications, I use the iOS client “MQTTool”. After signing up, head to “Subscribe” and type <code>birdnet</code> as
the topic and then click Subscribe. If everything is setup correctly and there are birds being recorded by the BirdNET-Pi’s
microphone, you should start seeing those detections in the MQTTool app. If so, fantastic news! Let’s move onto AppDaemon.</p>
<h2id="appdaemon-script">AppDaemon Script</h2>
<p>Now that we have the Pi communicating via MQTT, it’s time to get that information into HomeAssistant. I’ve shared <ahref="/posts/birdnet_homeassistant.html#birdnet-appdaemon-script">the full
script</a> at the bottom of this page, but let’s jump into each
section. This is not a full tutorial of how to use AppDaemon, but it may help fill in any knowledge gaps with the system.</p>
<h3id="imports">Imports</h3>
<p>First, we’re going to import <code>time</code> and <code>requests</code>. We’re going to use time as a backup to the <code>$time</code> component in the
payload. This can be helpful to see if there delays, or if BirdNET-Pi stopped detecting. We’re then going to use requests to
pull from Wikipedia’s API and grab a description for our HomeAssistant Dashboard.</p>
<h3id="class-definition">Class Definition</h3>
<p>To start any AppDaemon app, you need to include a Class that is defined in the <code>apps.yaml</code> file. This is also where we
initialize and define the various items that will be used in the remainder of the script.</p>
</span></span></code></pre></div><p>For this script, we need to use a lot of the AppDaemon APIs across more than just HomeAssistant, so we’re going to be using
<code>ADBase</code>. By using that, we can initialize the various APIs, which we do in the next 3 lines. In these 3 lines we need to get
access to HomeAssistant’s APIs, AppDaemon’s APIs, and MQTT APIs - the first and third items are plugins of AppDaemon, and
AppDaemon is… well… AppDaemon! Here are a few reference docs:</p>
<p>These will indispensable to you as you leverage AppDaemon and expand this little script.</p>
<p>Once we have access to that, we need to setup the main topic for MQTT from BirdNET-Pi and finally, what event we are
listening for that will trigger the functions in the rest of the script. <code>self.birdnet_mqtt = "birdnet"</code> is the definition
for the MQTT topic. Let’s breakdown the last line of the class.</p>
<p>Here’s a breakdown of each of the items in that last line. You can find the official documentation <ahref="https://appdaemon.readthedocs.io/en/latest/MQTT_API_REFERENCE.html#appdaemon.plugins.mqtt.mqttapi.Mqtt.listen_event">here</a>.</p>
<ul>
<li><code>self.mqttapi.listen_event</code> - this is what we use in AppDaemon to listen for an MQTT event in order to trigger a function.</li>
<li><code>self.birdnet_message</code> - the name of the function you’d like to trigger</li>
<li><code>"MQTT_MESSAGE"</code> - The default event in AppDaemon’s MQTT API plugin. This is used because MQTT doesn’t keep a state in this
plugin.</li>
<li><code>topic=self.birdnet_mqtt</code> - The topic that will be received to trigger the function. Defined on the previous line.</li>
</ul>
<p>In other words, what we are telling AppDaemon is the following: “When AppDaemon’s MQTT API plugin receives a message with the
topic of ‘birdnet’, run the function <code>birdnet_message</code>.”</p>
</span></span></code></pre></div><p>As such, we need to look <em>inside</em> the payload to begin grabbing the data. The <code>pre_split</code> variable is now just looking at the
data inside the payload and the rest of the variables take all the date into the payload, split it by the comma, and then
grab the string by their index. If you remember what <ahref="/posts/birdnet_homeassistant.html#birdnet-pi-notification-setup-mqtt">we did above</a> above, you’ll see that we have the various BirdNET information at each of the indexes in the AppDaemon script - 0 through 5.</p>
<p>This next section is shooting all the variables we just defined back via MQTT. The reason why we do it this way is because we
need HomeAssistant to grab each of these variables as individual sensors. BirdNET doesn’t give us that capability - it’s a
single message with all the information in one. <ahref="https://appdaemon.readthedocs.io/en/latest/MQTT_API_REFERENCE.html">Here is the documentation from AppDaemon</a> on <code>mqtt_publish</code>. Later on, I’ll show you how to ensure that HomeAssistant takes those topic payloads and adds them as
entities in your HA setup.</p>
<h4id="part-3-wikipedia-sensor">Part 3: Wikipedia Sensor</h4>
<p>The next eight lines are a fairly straightforward <ahref="https://en.wikipedia.org/api/rest_v1/">API call to Wikipedia</a>. We start
out by passing the <code>science_name</code> into the URL. The rest of the flags that we are passing into the URL comes from Wikipedia’s
<p>Once that’s done we call it with <code>response.get(url)</code> and format it with <code>response.json()</code>. Wikipedia returns the json payload
with the top level of <code>query</code> (which was our action in the url ;) ), and we’re looking for the value within that query.</p>
<p>All that’s left is to take that query value and push it to HomeAssistant! We can do that with the <code>self.hassapi.set_state</code>
function. Within the parenthesis we define the name of the sensor (<code>sensor.birdnet_wiki</code>), what it’s state should be (<code>on</code>),
and any attributes associated with the entity. Since we can’t assign a long description to the basic status of the entity,
we’re adding an attribute with the key of <code>description</code> and the value will be the wikipedia description garnered from the API
</span></span><spanstyle="display:flex;"><span><spanstyle="color:#66d9ef">for</span> value <spanstyle="color:#f92672">in</span> response[<spanstyle="color:#e6db74">'query'</span>][<spanstyle="color:#e6db74">'pages'</span>]:
</span></span></code></pre></div><h4id="part-4-generate-picture-for-detection-optional">Part 4: Generate Picture for Detection (Optional)</h4>
<p>This part is optional but I noticed that BirdNET-Pi was already grabbing a Flickr Picture for it’s front end, so I took the
code from the BirdNET code base and adjusted it a bit for my needs. This will work very similarly to the Wikipedia API call,
the main difference here being that you need an API key for Flickr. You can find more <ahref="https://www.flickr.com/services/api/misc.api_keys.html">information here</a>.</p>
<p>Given Flickr’s fairly robust API, by passing in the detected bird’s common name, we get amazing results from the community of
various pictures of the same species of bird. Ever since I’ve set this up, I’ve not seen a mislabeled picture in my
dashboard!</p>
<p>The most confusion portion of this section is the <code>image_url</code> as you’ll notice a bunch of <code>data["value"]</code> strings at various
portions of the URL. The short answer to this is in the previous line with the <code>data</code> variable. A successful query has Flickr
returning a large payload of information. We’re specifically using <ahref="https://www.flickr.com/services/api/flickr.photos.search.html">this</a>
Flickr API endpoint. While you can pass a lot of variables for your needs, if you scroll down, you can see that the example
response contains multiple photos in a single response. We’re passing <code>per_page=5</code> to limit some of those response items.
Left out of that response, though, is a one-stop-shop for a URL to that photo. Thankfully, Flickr can help us put together a
<p><em>Note: Full Transparency that I only learned about this after reading through BirdNET-Pi’s code base. Full credit goes to
<ahref="https://github.com/mcguirepr89">mcguirepr89</a>. For additional reference, here is Flickr’s <ahref="https://www.flickr.com/services/api/misc.urls.html">official page on construction
photo image URLS</a></em></p>
</div>
<p>With this response, we now have the variables we need to construct the URL to actually render the image. Those variables are:
Farm ID, Server ID, ID and Secret. I haven’t yet looked into why we need “farm” when the official documentation doesn’t state
anything about it.</p>
<p>Almost there! We now do the same as we did with the Wikipedia API response. We create a sensor in HomeAssistant! We’re
calling this sensor <code>sensor.birdpic</code>, ensuring the <code>state=on</code>, and giving it the attributes of the <code>image_url</code> as garnered
</span></span><spanstyle="display:flex;"><span> data <spanstyle="color:#f92672">=</span> flickr_resp<spanstyle="color:#f92672">.</span>json()[<spanstyle="color:#e6db74">"photos"</span>][<spanstyle="color:#e6db74">"photo"</span>][<spanstyle="color:#ae81ff">0</span>]
</span></span></code></pre></div><h2id="importing-mqtt-sensors-into-homeassistant">Importing MQTT Sensors into HomeAssistant</h2>
<p>Now that we have all the sensors defined and communicating via MQTT, we have one more step to import them into HomeAssistant.
<ahref="https://www.home-assistant.io/integrations/mqtt/">This MQTT documentation</a> by HomeAssistant is good to read about if you
need a broker setup. I will not be going over the broker in this tutorial, but may add one in the future. I tend to like the
yaml configuration for HomeAssistant, so for the sake of this guide, I’ll be referencing the <ahref="https://www.home-assistant.io/integrations/mqtt/#manual-configured-mqtt-items">manual configuration of MQTT
items and sensors</a>.</p>
<p>To add the sensors from above, open up your <code>configuration.yaml</code> file in your favorite editor. You’ll then want to add the
</span></span></code></pre></div><p>For the BirdNet sensors, we will be using a single domain: <code>sensor</code>. Feel free to copy and paste my config from below, but
make sure the names of each entity align with your needs, syntax, and nomenclature/system.</p>
<p><strong>Full MQTT Sensors in Configuration.yml</strong></p>
</span></span></code></pre></div><p>You might be looking at the list above and wondering where the Flickr and Wikipedia Description entities are. They were
already created by the AppDaemon script! Specifically, <code>self.hassapi.set_state()</code> function will either update the state for
an exisiting entity or, if the entity doesn’t exist, it will create a new one.</p>
<p>For the rest of the mqtt payloads, we need HomeAssistant to create them as they come in, which is why we add the above
code block to our HomeAssistant configuration file. To be clear, you <em>do not</em> need to add the Wikipedia and Flickr sensors to
HA’s configuration file!</p>
<h2id="adding-the-camera-entity">Adding the Camera entity</h2>
<p>Last but not least, we need to add a camera entity to ensure that the <code>sensor.birdpic</code> can actually be rendered visually.
It’s really easy to add this sensor, so this should be quick. Here’s how:</p>
<ol>
<li>In HomeAssistant, navigate to Settings > Devices & Services > Integrations</li>
<li>Click “+ Add Integration” in the bottom right-hand corner. Alternatively, if you already have a camera integration
enabled, look for the “Generic Camera” card and click “Add Entry”</li>
<li>Name the sensor. In this case, I called it “BirdPicturesfromFlickr” and renamed the entity to <code>camera.birdnet_flickr</code>.</li>
<li>You should now see a “Still Image URL” as the first of a few options on the screen. Enter the following into the still
image field: <code>{{ state_attr('sensor.birdpic', 'image') }}</code> (This is the sensor we created in AppDaemon with the flickr url
as the attribute).</li>
<li>Stream Source and RTSP transport protocol can both be left blank.</li>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-python"data-lang="python"><spanstyle="display:flex;"><span><spanstyle="color:#f92672">import</span> time
</span></span><spanstyle="display:flex;"><span><spanstyle="color:#75715e"># print(f"A {common_name} was seen on {date_seen} at {time_seen}. Confidence is {confidence}.")</span>
</span></span><spanstyle="display:flex;"><span><spanstyle="color:#66d9ef">for</span> value <spanstyle="color:#f92672">in</span> response[<spanstyle="color:#e6db74">'query'</span>][<spanstyle="color:#e6db74">'pages'</span>]:
</span></span><spanstyle="display:flex;"><span> data <spanstyle="color:#f92672">=</span> flickr_resp<spanstyle="color:#f92672">.</span>json()[<spanstyle="color:#e6db74">"photos"</span>][<spanstyle="color:#e6db74">"photo"</span>][<spanstyle="color:#ae81ff">0</span>]
</span></span></code></pre></div><p>To confirm that everything worked as expected, run <code>git remote -v</code> to check your remote repos. You should see one repo in
there twice, once for <code>(push)</code> and once for <code>(fetch)</code>.</p>
<p>I use the remote name “all” for multiple repos, so here’s what my <code>git remote -v</code> returns:</p>
</span></span></code></pre></div><p>To now push to your repositories, after adding and committing run <code>git push {{ remote-name }} --all</code>. My command is
<code>git push all --all</code> (see why I use all, now?)</p>
<p>Here’s the man page description on the <code>--all</code> flag:</p>
</span></span><spanstyle="display:flex;"><span> Push all branches <spanstyle="color:#f92672">(</span>i.e. refs under refs/heads/<spanstyle="color:#f92672">)</span>; cannot be used
</span></span><spanstyle="display:flex;"><span> Instead of naming each ref to push, specifies that all refs under
</span></span><spanstyle="display:flex;"><span> end, locally updated refs will be force updated on the remote end,
</span></span><spanstyle="display:flex;"><span> Do everything except actually send the updates.
</span></span><spanstyle="display:flex;"><span> same as prefixing all refs with a colon.
</span></span></code></pre></div><p>And that’s it! You should be able to push everything to both of your repos fairly easily now with this new set commands.</p>
<p>For the last 3 days, I have been spending a few hours after working trying to figure out why my brand new Hugo site was not
loading correctly on my sub-domain. For context, I use Nginx to host all my apps and servers, most of them using reverse
proxy protocols such as <code>$proxy_host</code>, <code>$forward_scheme</code>, and <code>$port</code>. There are a few more and I’m happy to share some
reverse proxy nginx config files. See my post on <ahref="/posts/npm_to_nginx_tutorial.html">moving from NPM to Nginx</a> for more information.</p>
<p>Once I got the basic idea of this blog up and running, I ran <code>hugo</code> and then used <code>scp</code> to send the files to my nginx host’s
public folder. Despite <code>index.html</code> and all the CSS files being in the right spots, I kept getting a few repetitive errors
from nginx - either in my browser’s console or in nginx’s log files. I swore I read through every StackOverflow or personal
blog post I could find. In fact, the next day, I would sit back down after work to debug and I would continue to find the
same sites and posts I found the day before! It was getting frustrating.</p>
<p>Wouldn’t you know… it was one of the simplest solutions that got it all working. Here’s a breakdown of what I was seeing
and my hypothesis.</p>
<h2id="errors">Errors</h2>
<p><em>Console Errors:</em></p>
<ul>
<li>Incorrect MIME type –><code>css</code> files being set as <code>text/html</code> type.</li>
<li>500/502 Errors when trying to load javascript files</li>
<li>500 Errors when trying to load child pages.</li>
</ul>
<p><em>Nginx Log Errors for this server:</em></p>
<ul>
<li><code>[error] 1147432#1147432: *84013 invalid URL prefix in "://:/favicon-16x16.png"</code></li>
<li><code>[warn] 1147432#1147432: *84013 using uninitialized "port" variable</code></li>
<li><code>[error] 1118832#1118832: *77105 directory index of "/var/www/html/" is forbidden</code></li>
</ul>
<p>I thought I had tried everything, but it was a rip and replace from a single blog post that solved it for me. Some of the
things I tried include the following. _Note: when I mention ’nginx config file’ below, I am referring to the specific file
for this subdomain. I have a single global nginx.conf and then individual files for each subdomain/proxy host.</p>
<ul>
<li>Editing <code>index.html</code> to ensure that any referenced file had an explicit mime type associated with it.</li>
<li>Included <code>include { full path }/mime.types</code> in the specific nginx config file.</li>
<li>Included specific <code>location ~ \.(css|js)$ {</code> sections in my nginx config file.</li>
<li>Tried assigning the <code>$forward_scheme</code>, <code>$host</code>, and <code>$port</code> variables (similar to a reverse proxy host).</li>
<li>Removing any SSL references <– This caused similar behavior but different errors! I thought I was making progress…🫥</li>
<li>Switched out the variables of <code>root</code> and <code>alias</code> between my <code>server</code> and <code>location</code> blocks.</li>
<li>Started with something super simple, such as the suggestions from <ahref="https://gideonwolfe.com/posts/sysadmin/hugonginx/">Gideon Wolfe’s Block</a>.</li>
</ul>
<p>If you clicked on the link I just shared, you’ll see that Gideon’s setup is quite similar to the one that I finally got to
work. My thought there is that my errors were less about the nginx config setup from within the file, and instead it’s very
possible that I set incorrect directory permissions after transferring all the public files to the web server. Gideon’s blog
was the very first I clicked on, so I owe them my thanks since their site was the entry point to figuring all this out!
You can find a list of all the blogs I stumbled upon in this weird and fluctuating journey of doing something as simple as
sharing my static files on an nginx web server.</p>
<h2id="solution">Solution</h2>
<p>In the end, <ahref="https://newbs.rocks/posts/hugo-setup/">newbs.rocks blog post</a> on setting up Hugo with nginx provided me a config
example that worked for my setup. I think part of what happened was that as I was cycling through all the blog posts and
StackOverflow posts, I would remove one or two lines (usually the one or two I changed from the previous post’s
recommendations) but in doing that, was making more of a mess for myself, burying the error even more deeply. By replacing
everything, I’ve brought it back to a manageable place.</p>
<p>You’ll also notice in my <ahref="/posts/hosting_hugo_troubles.html#nginx-config">final config file</a>, I was able to add back in the <ahref="https://www.authelia.com/">Authelia</a> snippets, paths for the SSL certs, and a few other items that connect nginx to the rest of my infrastructure.</p>
<p>If you’ve stumbled upon this, I hope it helps you figure out your Hugo/Nginx issues! I definitely saw a lot of people posting
in Hugo’s Discourse asking about <ahref="https://discourse.gohugo.io/search?expanded=true&q=mime%20type">mime type errors</a>, so it is
very likely that whatever you’re facing isn’t isolated to just you.</p>
<p>Now that I have this up and running, I need to write (and post!) a script that will pull from my repo, change directory
<p>A Tutorial Repo for migrating your Nginx Proxy Manager proxy setup to Nginx. I wrote this originally for <ahref="https://www.reddit.com/r/selfhosted/comments/15j4v80/minitutorial_migrating_from_nginx_proxy_manager/">this reddit
post</a> and to post this <ahref="https://github.com/Normanras/Npm_to_Nginx">my Github profile</a>. Thought my website would also be a good place to share it for any passers-by.</p>
<p>To give clear instructions to help users migrate from using <ahref="https://nginxproxymanager.com/">Nginx Proxy Manager</a> (NPM) to standard <ahref="https://docs.nginx.com/">Nginx</a>. This tutorial is not exhaustive and there are many other implementations of this transition. I would recommend checking out the many Nginx <ahref="http://nginx.org/en/docs/">Documentation Sites</a> and tutorials to learn more.</p>
<h2id="introduction">Introduction</h2>
<p>If you’re anything like me and you got into the self-hosted/homelab/diy game sometime within the last 5 years, you’ve likely been recommended to use Nginx Proxy Manager as one of the choice Reverse Proxy services. If you’ve also been paying attention to various self-hosted communities, you may have also come across Christian Lempa’s Video on <ahref="https://youtu.be/uaixCKTaqY0">trusting smaller self hosted projects and tools</a>.</p>
<p><em>Spoilers:</em> He roasts NPM in his video and towards the end says he won’t be using NPM anymore. He also, perhaps purposely, doesn’t share which tool he will be migrating to.</p>
<p>Whether you follow Christian away from NPM or not, it dawned on me that while NPM is using a very trusted web server and reverse proxy under the hood, I hadn’t taken the time to understand how an Nginx Config actually worked. Since NPM was already creating most of the files for Nginx, I got to reading through all the files and reworking them so that I could begin using Nginx without the NPM gui.</p>
<p><em>Contributing: This is not all encompassing of Nginx possibilities. Including instructions for various installation methods, using OpenResty, and any other migrations or use cases would help the community. If you’d like to add in additional information on how to migrate from NPM to Nginx, that is welcome. Simply submit a PR with your steps.</em></p>
<p>Copy the following contents (including sub-directories) from the NPM <code>/data/nginx</code> directory to the Nginx <code>/etc/nginx</code> folder:</p>
<p>Edit each file in your <code>sites-available</code> directory and update the paths. Most will change from <code>/data/nginx/</code> to <code>/etc/nginx</code>.</p>
</li>
<li>
<p>Edit your <code>nginx.conf</code> file and ensure the following two paths are there:</p>
<ul>
<li><code>include /etc/nginx/conf.d/*.conf;</code> and <code>include /etc/nginx/sites-enabled/*;</code></li>
</ul>
</li>
<li>
<p>Symlink the proxy host files in <code>sites-available</code> to <code>sites-enabled</code></p>
<ul>
<li><code>ln -s * ./sites-enabled</code></li>
</ul>
</li>
<li>
<p>Test your changes with <code>nginx -t</code>. Make appropriate changes if there are error messages.</p>
</li>
</ol>
<h2id="pre-requisites--assumptions">Pre-requisites & Assumptions</h2>
<p>I am using an Ubuntu VM with NPM and it’s db as a Docker Container while Nginx is installed natively on the machine. You don’t have to use this setup exactly, but I am making a few assumptions as to what you should have access to before you begin. I am also using custom SSL certs, but theoretically, the transition should be the same when using Lets Encrypt.</p>
<p>I’ve added some example files to show before and after changes to this repo and outlined file trees below.</p>
<ul>
<li>You understand the basics of what a Reverse Proxy is doing and are sticking with some stock settings (like exposing port 80 an 443).</li>
<li>You’ve installed <ahref="https://nginxproxymanager.com/setup/">NPM</a> and <ahref="https://www.nginx.com/resources/wiki/start/topics/tutorials/install/">Nginx</a> using your preferred method.</li>
<li>You have access to both NPMs file tree and Nginx’s.</li>
<li>If using NPM in docker, make sure you’ve mapped a local volume on the host to the container.</li>
<li>My setup using docker-compose is the following: <code>/user/nginx/data:/data</code>.</li>
<li>Know where your Nginx files are. If using docker, same as above, make sure your container directories are mapped to the host.</li>
<li>For a linux install, they should be accessible at <code>/etc/nginx</code>.</li>
<li>You know how to edit files at the command line using <code>nano</code>, <code>vi</code>, <code>vim</code>, <code>neovim</code>, <code>emacs</code>, or something else.</li>
</ul>
<h2id="nginx-files">Nginx Files</h2>
<p>Nginx uses the <code>nginx.conf</code> file and within that file, it will include your proxy files. These exist under <code>./nginx/sites-enabled/</code>. In the main <code>nginx.conf</code> file, the line <code>include /etc/nginx/sites-enabled/*;</code> will bring in those files to the config file, making the proxies accessible.</p>
<h2id="how-to-transition---detailed-version">How to Transition - Detailed Version</h2>
<ol>
<li>
<p>Since NPM uses Nginx under the hood, they are both, by default, going to try and use ports 80 and 443 to serve up your apps and content. Turn off both systems.</p>
<p>Nginx doesn’t really care what the files are called, but NPM numbers them based on the order in which you added them in the GUI. I find it better to rename them to what service they actually serve up for easier identification later.</p>
</li>
<li>
<p>Copy your <code>custom_ssl</code> folder from NPM to the <code>custom_ssl</code> folder in Nginx. See the following step for the various default paths in both systems.</p>
<p>Copy the <code>conf.d</code> folder from NPM to the <code>conf.d</code> folder in Nginx. <em>Note: For some reason, not all of the files in
the proxy files were actually in my <code>conf.d</code> directory. If you’re missing any files, please download and/or copy and
paste them from the <ahref="https://github.com/NginxProxyManager/nginx-proxy-manager/tree/fa851b61da3fe3726d1a04c25e69d36e79edea2d/docker/rootfs/etc/nginx/conf.d/include">NPM Repo</a></em></p>
<p>If you had any additional files included in the Advanced section of an NPM Proxy Host, make sure you copy them over. For my setup and this tutorial, they were all located in the <code>snippets</code> directory.</p>
<p>There are a number of lines that need to be updated in each proxy configuration file to make them work with Nginx. I’ve placed additional comments in <ahref="./proxy_host/npm_proxy.conf"><code>./proxy_host/npm_proxy.conf</code></a> file. The line changes are the following:</p>
<li>The paths should remain the same. However, if you changed the path for <code>conf.d</code> in Nginx and differed in step 5, above, make sure you use the correct path.</li>
<p>Double Check all your paths! If this is your first time using Nginx, make sure every directory is correct! Save your work.</p>
</li>
<li>
<p>Navigate to the <code>nginx.conf</code> file which is located at <code>/etc/nginx/</code>. You can see the one I am using in this repo.</p>
</li>
<li>
<p>Make sure that you have the following two lines in the main conf file and that they are pointing to the appropriate directories in the nginx directory path.</p>
<ul>
<li><code>include /etc/nginx/conf.d/*.conf;</code> and <code>include /etc/nginx/sites-enabled/*;</code></li>
</ul>
</li>
<li>
<p>You’ll notice that you ensured there was a <code>sites-enabled</code> directory in the configuration file, but you changed all your proxy host config files in <code>sites-available</code>! Good eye, all that’s left is to symlink the files to <code>sites-enabled</code> so that nginx can start using them.</p>
</li>
<li>
<p>To symlink the available proxy files run the following command within the <code>sites-available</code> directory:</p>
<ul>
<li><code>ln -s * ./sites-enabled</code></li>
</ul>
</li>
<li>
<p>Once you’re confident that you’ve done all the above correctly, you can test your setup using nginx command and flags. While in the directory with your <code>nginx.conf</code> file - usually <code>/etc/nginx</code> - run the following command: <code>nginx -t</code>.</p>
</li>
<li>
<p>If all is working as expected you should see the below output. If it returns any errors, fix them appropriately. It will usually tell you what line is throwing the error. In this case, there’s a high likelihood that it will be path error.</p>
</li>
</ol>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-bash"data-lang="bash"><spanstyle="display:flex;"><span>nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
</span></span><spanstyle="display:flex;"><span>nginx: configuration file /etc/nginx/nginx.conf test is successful
</span></span></code></pre></div><p>And that’s it! You can now restart your nginx service on the host and access all your sites just as if you were using Nginx Proxy Manager! Make sure you take a look at your logs and system’s status should nginx fail to start.</p>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-bash"data-lang="bash"><spanstyle="display:flex;"><span>├── data
<content:encoded><