Combining two log files into one using PowerShell

04 February 2016

Here’s the more or less finished script from my previous post.

My IRC logs are structured into folders like this:

irclogs_old/
  UnrealTournamentNet/
    #game.log
    #catpix.log
    query.log
  ChatNet/
    nickserv.log
    #erp_chat.log

------

irclogs_new/
  UnrealTournamentNet/
    #newgame.log
    #catpix.log
    friendguy.log
  Horison/
    #news.log

Because of this, the script needs to append new log files into old ones if they exist (#catpix.log in this example), create a new log file if an old one doesn’t exist, and also create new folders for networks that don’t have old logs but have new ones.

It works for my use case, but if the folder structure is any deeper than this the script needs to be modified. Replace 1 and 2 on the first two lines with the names of the folders your old and new logs are in, respectively.

$origFolder = join-path $pwd "1"
$newFolder = join-path $pwd "2"

Get-ChildItem -recurse -include "*.log" -path $newFolder |
ForEach-Object {
    $folderPath = join-path $origFolder $_.DirectoryName.Split("\")[-1]
    $newPath = join-path $folderPath $_.Name
    if((Test-Path $folderPath) -eq 0) { mkdir $folderPath }
    Add-Content -NoNewline -path $newPath -value (Get-Content $_ | Out-String)
}

It takes the logs from folder 2, and appends them after logs in folder 1. If a file doesn’t exist it should be automatically created by Add-Content, but the command can’t create folders so we need to check for that separately. It only goes one folder deep, so it needs to be modified if the folder structure is different.

You’ll probably want to backup your logs before using this just in case. It took a couple of minutes to go through my 700MiB of logs. Odds are it could be optimized to be faster but whatever.

Combining two IRC-logs into one with PowerShell: A Cautionary Tale

03 February 2016

I have a reliable Unix shell that I use for running irssi (an IRC client) over SSH. The caveat is that the server the shell is on uses Very Expensive SCSI hard drives, which is why they can only afford to give users 512MB of disk. Or that’s what the sysadmin tells me, at least. Either way, it’s 20 eurobucks a year and very stable so I’m content.

This small quota tends to fill up rather quickly with logs from the 20 or so channels I’m on. Thus I must periodically delete the logs or else I’ll run out of disk space. Whenever I do this, I end up with two separate log files: the old one which I’ll download and store locally, and the new one which starts slowly rebuilding on the server.

Because I’d prefer to have one uniform logfile that contains everything that ever happened on that channel when I was on it, every time I reset the logs I need to append the new logfile to the old one. A very simple task, but strenuous to do by hand for dozens of channels and queries.

So for the longest time I’ve had a task on the backburner: write a script that automatically appends new logs to old ones. This, too, is a simple task. Or should be. One line in bash and you’re done.

But I’m on Windows. I don’t have Bash, and I don’t want to install Cygwin because I’ve been told it’s shit and don’t care to find out if this claim is true.

Windows is in Microsoftland, and in Anno Domini 2016 inhabitants of Microsoftland use Microsoft(r) PowerShell(tm). Off I went to figure out how to write an extremely simple little script in PowerShell, of which I have zero prior experience.

An hour and dozens of Duckduckgo searches later I emerge from underneath a million StackExchange tabs, carrying with me the scripture:

$origFolder = join-path $pwd "1"
$newFolder = join-path $pwd "2"
$encoding = "default"

Get-ChildItem -recurse -include "*.log" -path $newFolder |
ForEach-Object {
    $newPath = join-path (join-path $origFolder $_.DirectoryName.Split("\")[-1]) $_.Name
    Out-File -filepath $newPath -Append -NoNewline -Encoding $encoding -InputObject (Get-Content $_ | Out-String)
}

The best part: it actually works. Kinda. Except it doesn’t. It ruins character encoding, and I can’t for the life of me figure out why.

My IRC logs are encoded in UTF-8. If I tell the script to use UTF-8 encoding, special characters like ä and ö become garbled within the UTF-8 encoded IRC log file created by irssi on the unix server. However, in a test file created locally that also uses UTF-8 they’re completely fine.

On the other hand, if I tell the script to use the “default” encoding (which “uses the encoding of the system’s current ANSI code page.”, whatever that is), the IRC log file turns out fine while the test file becomes garbled.

After some research I discover an alternate command for appending to files. This command doesn’t care about character encoding – I don’t know how it can do this when encoding is such a big deal to the other one. I replace the Out-File line with this:

Add-Content -NoNewline -path $newPath -value (Get-Content $_ | Out-String)

Miraculously, both the real logfile and the test logfile pass through unscathed! It seems that the script is actually working now. Further testing is required, but I’ll leave that for tomorrow.

P.S. Note-to-self: get a blog theme with better looking code tags.

On MMORPG character creation

02 February 2016

Some of my friends are excited about an upcoming MMO called Black Desert Online. I’ve known about it for a long time, but I know better than to get excited about new MMO releases. That’s a good way to feel very disappointed.

The game is launching in March, but they’ve made the character creator public for people to toy with. If their marketing is anything to go by, they treat the character creator as their pride and joy and think it’s really something special. The (carefully edited) preview video sure tries its hardest to give this impression:

After trying the editor yourself it’s painfully obvious the video has been meticulously cut in addition to just being sped up. In practice the character creator is a humongous mess of sliders, which fortunately can slightly more intuitively be manipulated by dragging your mouse on various parts of your character’s face like in the video. Such freedom! It’d be easy to think that this is one of the best, if not the best, character creator in an MMORPG or any game so far.

Alas, that freedom is the greatest fault in the creator. Basically what it does is exposes every single parameter in the underlying face structure engine to the user, with little to no limits or moderation, in a somewhat intuitive manner. The implementation is admittedly quite slick, but as a concept it’s hardly new, with many singleplayer RPGs already having used the exact same trick before. Dark Souls 1 and 2, Oblivion and Skyrim (with mods), and Dragon’s Dogma to name a few off the top of my head.

Because there are no ‘sanity checks’ of any kind on the sliders, it’s a cinch to make your character abhorrently ugly, be that on purpose or not. A single click and drag can make your character’s eyes clip into their cheeks or pull their lips through their teeth. With some fiddling you can push the eyeballs out of their sockets, shape lips into some kind of pointy beak parts, and top the whole thing off with heart-shaped pupils and optic fiber afro:

Creating actually good- and most importantly unique-looking characters is much more difficult. Expect to spend several hours in the editor, and when you finally think you’re done and create your character, sooner or later while playing you’ll notice some little detail that’s off and will have to pay real money to re-edit your character to fix it. That’s how it usually goes for me, at least.

As I said earlier, it’s easy to fall into thinking that this is a good character editor. A lot of people would say it’s on a whole new level. I disagree. I think it’s lazy fanwanking.

Read more