Welcome to my homepage

Best Viewed Without Eyes

Migrating MySQL to PostgreSQL with pgloader

I've been fighting with getting a database moved from #MySQL to #Postgres for about a week now because of a dumb issue with Directus, and I finally fucking got it. The trick was, I think, to just make pgloader run slower.

I tried using the default settings that the pgloader docs suggest, only to have it shit the bed over and over at different spots, and throw errors about not being able to connect to either the PG server or the MySQL server, or simply hanging until I ctrl-c (or kill -9 sometimes -- it's a bastard) the fucking thing.

Anyway, here's the load file I ended up using.

LOAD DATABASE
  from mysql://username:password@host:port/database
  into postgresql://username:password@host:post/database

 WITH include drop, create tables, create indexes, reset sequences,
   workers = 1, concurrency = 1,
   multiple readers per thread, rows per range = 1

  SET PostgreSQL PARAMETERS
    maintenance_work_mem to '128MB',
    work_mem to '12MB',
    search_path to 'database, public, "$user"'

  SET MySQL PARAMETERS
    net_read_timeout  = '120',
    net_write_timeout = '120'

 CAST type char when (= precision 36) to uuid drop typemod

Slap that shit in a file (obviously replacing all the necessary bits with your own stuff), and the fire it up with pgloader like:

$ pgloader data-loader.load

Then sit back and wait for your database to migrate :-)

I hope this helps someone in the future, because the documentation, and the --debug output of pgloader didn't do me a damn bit of good. It was by pure luck and chance that I thought "let's try cranking that workers and concurrency down, and that rows per range, too, and see what happens. Well la-dee-fucking-dah, it worked.

Fun side note: Since this was for Directus, of course it presented another set of challenges once the database was migrated. Starting the server resulted in a Database doesn't have Directus tables installed error with zero other details. Turning the LOG_LEVEL to trace shed a little more light, which led me to this comment on GitHub. A tweak to my .env, like so:

DB_SCHEMA="my database name"
DB_CUSTOM_SCHEMA="my database name"
DB_SEARCH_PATH="array:my database name,public"

And 💥 the server runs!!

Fuck you, computers. Fuck you to death.

Setting multiple variables with a ternary in JS

There's no way this hasn't been documented somewhere already, but it just dawned on me last week, and I thought it was pretty neat!

With the fun power of destructing arrays in #javascript, you can set multiple variables in one go using a ternary expression. Check it.

const [var1, var2, var3] = someVar
  ? ['true value 1', 'true value 2', 'true value 3']
  : ['false value 1', 'false value 2', 'false value 3'];

Not too bad, eh?

NodeJS Podcatcher

I've been using Downcast for a while, but it's got some really annoying bugs that I just can't deal with any more. It's basically just a podcatcher - a very slow podcatcher - at this point, and it constantly eats up about 50% of two CPUs when it's doing nothing, and even the podcatching is painfully slow.

So I wrote some javascript (with a little help from ChatGPT, because why not) and now I can download podcasts when I want without eating a bunch of CPU and being dumb. Yay!

Shared: A Wooden Artwork Miraculously Unfurls into a Functional Desk Designed by Robert van Embricqs — Colossal

Some very slick wood working. HT Jason Kottke

A Wooden Artwork Miraculously Unfurls into a Functional Desk Designed by Robert van Embricqs

That didn't take long

Today while throwing the ball around with the big dog, I had the thought that I should completely rewrite this thing in #NextJS, just because. I haven't followed through on it... yet.

It could be fun, I suppose. But I also have other things that I know will be fun and will not, at any point, make me question every decision that led me here. I could spend an entire day playing #guitar, getting blisters on my fingers, and making exactly zero #music, and still probably have just as much fun - if not more - than sitting here, banging away on this fucking keyboard writing code that'll be used by zero people. Not counting me, of course :-)

I finally made the code for this dumb thing public

I'm not sure why I even bothered since I'm completely positive I'll be the only person that ever uses it (and even me, I'm not sure how much longer, because who doesn't like reinventing stuff over and over again for absolutely no reason?) but also, why the hell not, I guess? Doesn't the world need more bad #opensource #php code? Do you want a dumb website like this? Go grab the code, and when it doesn't work, make an issue (or better yet, a pull request!) over on Github :-)

Covid 2019, 2022 edition

Got #boosted yesterday at 11AM. First half of the day was fine, but around 6PM, things went downhill fast. Spent the rest of the evening in and out of sleep on the couch until 10:30, then went to bed. I was freezing. Woke up every couple hours for some reason - sweating, freezing, rolled over on my arm, you name it. It's all done now, though.

I slept for nine-and-a-half-hours according to SleepCycle, and got 100% "sleep quality," and I'm feeling much better today. Not back to 100%, but probably somewhere around 85%. I can't imagine what actually catching #covid is like, especially without being vaccinated. If it's anything like what I've gone through (for every fucking shot), then I'm happy to deal with it for half of a day and be done with it. And hopefully I can continue to avoid catching it. 🤞

Daily recap - 2022/09/02

We went on a #hike in Garden of the Gods after business pizza. That was fun. We saw at least 6 deer, maybe 8. Missed a pretty great sunset by about 30 minutes, but it was still a good hike. Lauren got to break in her new hiking shoes a bit, and I got my "exercise" in without having to walk on the fucking treadmill.

Work has been #work. Nothing fun, nothing terribly exciting. Day job has turned into just doing billing stuff - new prices, new plans, new subscription types, accepting payments from other countries, implementing 3D secure... Billing, billing, billing. I've been working on a desktop app (in #Electron, of course) for the night job. Not super fun, but it's different at least, which is kinda fun. Though I question the reason for its existence... I have a feeling that, with a bit of #CLI expertise, the person requesting this application wouldn't need it, but who am I to deny someone paying us to write code for them?

Storm's coming. I've been watching the lightning for the last half hour. I can finally hear the thunder. I hope it's a good one. It'd be nice to go to bed to some big ol' fat rain drops hitting the window.

Daily recap - 2022/08/02

Jack shit going on. Back to work this week after a week off. My nephew came to visit and I took some time off so we could do fun stuff. We went up to the top of Pikes Peak, walked around some downtown spots, ate out a bunch, saw Nope, played video games at home and in arcades, and just had a good time. The best part though, was sleeping in. Good fucking christ do I miss sleeping in. Not until like, noon - which I could definitely do - but like, just 10am. There's nothing stopping me from doing that now, other than the fact that I'd only have an hour between jobs, which is a pretty big deterrent. Ugh. Maybe one day I'll be able to just work one job 🤞

Daily recap - 2022/07/20

Day job, night job. Food between. Nothing special happened today. But night job lately... Oh boy.

I've been tasked with building something to let people see when there have been changes made to a website. A new page, a change to a page, whatever. That sounds easy, right? Just scrape some pages, save the text, and the next time you scrape the page, compare the new text with the last bit of text we saved, and if there are differences, save those. Easy peasy, right? Fucking wrong!!

First off, scraping a site - by which I mean using some code to "look at" the homepage, grab the content and save it, extract any links in that content, figure out if we should follow any of those links, then follow them and start the whole "extract content, find links, follow" process until there's nothing else to "look at," is no easy task. It sounds simple, but it's really not. One way we can do this is to grab a page and try to extract links with regular expressions. That's a bad idea for a number of reasons that need no explanation. Another better option we have is to use a library, like Cheerio, that does all that hard work for us and let's us do something like $('a[href]') to pull all the links out of a page, but we still need to figure out if they're links we should follow. And if humans had anything to with the composition of a page, there's a good chance you'll encounter some dumb shit that makes no sense and needs a special exception in your code. Broken HTML? Guaranteed. Links with href values that make no sense, but a browser can somehow understand? Sure. Links with no href when the HTML loads, but they get populated by some Javascript after the fact. That wouldn't surprise me one fucking bit. All this is to say absolutely nothing of the fact that a lot of sites, especially larger ones, have protections in place to prevent bots from doing exactly the thing I'm trying to do.

Okay, so "scraping" a site is out of the question. Well, you know what most sites (though, not the bigger ones, strangely enough) have? Sitemaps! Boom!! We'll just grab the sitemaps, which tell us exactly which pages exist on the site, and grab the content on those pages. Easy enough. Oh, except some sitemaps contain URLs that aren't valid. https://somedomain.com /a-page.html is not a valid URL, unless I missed the announcement from ICANN that "dot-com-space" is a new TLD. I guess technically https://somedomain.com/./a-page.html is a valid URL, but like, what? Slash-dot-slash? Was this website built by Fatboy Slim? Oh, and these sites will also block us if we scrape pages too quickly. OH! Some sites have thousands - or hundreds of thousands - of pages. So even with a one second delay between scraping pages, it'll still take a stupid amount of time to scrape an entire site.

Oh, you know what?! robots.txt tells us pages we're not allowed to scrape, and also (I learned during all of this) how long we should wait between scraping pages. Let's look at the robots.txt and make sure (a) we're not scraping stuff we shouldn't and (b) we're not scraping a site too quickly. Of course, any jackass would realize that robots.txt probably isn't telling them what not to scrape or how long to wait 100% of the time, but I'm a special kind of jackass, and that did not occur to me. Not every robots tells you how long to wait, or what you're not allowed to scan. Cool, cool, cool, cool.

So here I am. Waiting sometimes 4 days for a site to finish being scraped (I'm not sure how long it actually took because I killed the process after 4 days) and learning about all the ways people can supremely fuck up something as simple as a sitemap. Seriously?! SLASH-DOT-SLASH?! WHAT THE FUCKING FUCK?!. I'm pretty sure the only way to get this thing to run in any sort of timely fashion is to completely rewrite it so it can be distributed across a fuckton of servers and processes... And a lot more infrastructure.

Daily recap - 2022/07/16

Nothing happened today. Saturday errands, then hanging out writing #code. Code writing went well, but Github is still being a fucking asshole.

I'm not sure what the problem is, but I've got a test that consistently fails at this one spot in their "Actions," but it runs just fine on my machine. Ugh. It's gotta be something with how I've got the action configured, or maybe the code is legitimately broken, or maybe it's something else, but I'm having a hell of a time figuring out which of those things it is.

One very interesting thing - during my last session of test debugging, I realized my local Redis version didn't match the version in my workflow config, so I updated the workflow config to match... And the fucking tests PASSED! Literally all I did was change the Redis version in my workflow config, commit that, and boom, I got the green check. "No fucking way" I thought, and I was right! Because on the next commit where I just removed some caveman debugging, the goddamn tests failed. Ugh.

No clue what the problem is, but I've decided I'm not too bothered by it. The tests pass for me (classic "it works on my machine 🤷‍♂️") so I'm not going to lose any time trying to figure out why Github hates me so much. I can still write code and run tests, so suck it, Github... Suck it dry.

Weird car in Colorado Springs

Lauren and I were out running our Saturday morning errands, and we pulled into Ace to discover this beast.

The owner wasn't around so I couldn't find out anything about it, but just look at this thing! It looks completely custom, and completely bonkers. I wish I could've talked to the person that built it and taken a ride... It is probably one of the scariest and most fun things in the world.

Daily recap - 2022/07/07

Found this today after thinking "So many song lyrics are just nonsense, I bet I could use GPT3 to generate lyrics" and then thinking "I bet someone else has already done this." Not great, but also, not the worst. https://www.jarvis-lyrics.com/

Finally finished helping my friend get some servers moved around. The database was the worst. Some bullshit with needing permissions just right because computers are stupid... Anyway, it's done now. Hopefully I won't have to fuck with it ever again.

Daily recap - 2022/07/05

Nothing today. I'm exhausted. Went to bed too late last night. But yesterday, holy shit.

A dude across the street blew up his hand! Like, people-looking-in-the-street-for-parts-of-it blew it up. Of course I helped look, who could resist?! I saw what people said was part of a thumb. A girl pushed it around on the pavement with her foot. I assume it was a part of something because someone quickly came and collected it in a bag of ice and rushed it off to the hospital. And then there was a fist fight at the same house. Nobody called the cops (good). I'm pretty sure the sun hadn't even set.

After that crazy shit, nothing eventful. Friends, fireworks, beer, food (so. much. food! ribs, chicken, coleslaw, mac-n-cheese, potato salad, baked beans, potato wedges, candied bacon, and more stuff that I'm forgetting - Lauren fired up the grill at 10AM) music, lasers -- one of our friends is a DJ and he setup some speakers and a laser projector on our house -- it was pretty rad.

The neighbors seemed to enjoy it, and one guy even came over and thanked us for the entertainment. The fireworks in the street, that have happened the past two years, pretty much stopped after that dude fucked himself up, but there was still a great show in the park and literally all around us in the neighborhoods. Overall not a bad night. For me, anyway. I'm sure that dude with the messed up hand has different feelings.

Daily recap - 2022/06/26

Doing this early today because instead of spending my whole damn day in my office, I'm going to setup the HTPC and hang out in the living room.

Today was good. Talked to both parents for a little while and got some another Indieweb feature - Webmentions - implemented on this dumb thing.

My new keyboard (Keychron K8 with hot-swappable keys) arrived yesterday, and I'm quite pleased with it. It feels a bit quieter than my previous keyboard, though that could be due to the desk mat that I also purchased. Regardless, it's nice to have function keys and number keys and to have the arrow keys back in a place that makes more sense... Now I just need to learn some new muscle memory for those things since I got so used to where they were on my last one. This week will have more typos than usual, I'm guessing.

We also got 3 new pieces of furniture - a credenza for the dining room, an entertainment/tv stand thing for the living room, and the bonus piece, a VHS/CD cabinet that Lauren is using to store her video stuff - for the price of 2! When we were picking up one piece, the seller offered us another one for free. Not too bad at all.

Back to the grind tomorrow. And then only a week until we get a fireworks show on our street.

Daily recap - 2022/06/24

Well, this happened. Can't say I'm surprised after the leak a while back, but it's still pretty amazing.

Nobody was stopping people from not getting abortions, so I don't understand what the problem was. What is the deal with these fucking people? Why do they want to impose their beliefs on everyone? Why do they think that making someone have a kid they don't want (for whatever reason) is going to be a good thing for the kid or the person that's gotta raise the thing? Why do they want to give everybody guns? Why do they get elected in the first place? What the fucking fuck is going on?!

In the car today I heard the president on the radio saying "Roe is on the ballot in November. You've gotta elect people that will..." blah blah blah. Dude, we elected you!! What are you doing? There's no shortage of democrats in power, and there's a democrat in the fucking white house! Executive order that shit or something, I don't know. The last dude didn't seem to have a problem doing that, why can't you?

Anyway, nothing happened for me today. Errands and yard work tomorrow. Might be picking up a couple bits of furniture I found on Facebook... We're at least gonna go take a look at 'em.

Daily recap - 2022/06/23

Another (almost) week of work, another week of nothing really exciting. We went to a Rockies game on Sunday for Fathers Day with Lauren's uncle, and that was pretty fun. Expensive as all fuck, but fun.

I'm beginning to regret this whole "daily recap" thing, since most of the time it's just me talking about what I had for dinner and absolutely nothing else... Whatever.