Expanding Our On-Line Footprint

Programmer At Work

Programmer At Work

Over the past few weeks I’ve performed a major upgrade to our small home on the Internet in the form of a genealogy section. This is a project I’ve been wanting to do ever since I re-launched this website, and finally took yet another plunge. It’s already turned into a big project, but one that’s well worth doing.

I was inspired in part to launch this new site because a few months ago I received several boxes of Dad’s papers from my brother Howard. Howard had been storing Dad’s belongings ever since Dad passed in February 2004, and this past summer finally got around to going through it. He discovered a lot of genealogy material that Dad had collected in his last years, really good information that took a lot of research.  I have also amassed a lot of old photographs and family information over the past years from other relatives and really want to share it on-line.

The genealogy site is a proprietary software package named “The Next Generation of Genealogy Sitebuilding” (TNG), and has been around for at least 15 years. It’s probably one of the better genealogy packages available for small sites like this one. I had used an earlier version of this software on the previous PHPNuke powered website so I went into this project knowing what I was getting into. Which is different than most projects I launch.

The TNG software is built as a stand-alone website system, but does offer the ability to integrate it into other popular open source packages including WordPress. At least in theory. The earlier incarnation offered the ability to integrate into PHPNuke, but it did not integrate well. Given how flexible WordPress is I’d hoped to have more luck this time around.

I purchased the package and performed the down load about one month ago, and that was the last part of the installation to go well. First up, I had to upload the several hundred TNG files to my server. The normal utility to use for this is FTP, but the last time I used my FTP software was before I migrated to my present host. After a week of screwing around trying to figure out FTP settings, I discovered my host had a built-in utility for uploading mass quantities of files. I cheerfully dumped my FTP utility.

Next up was performing the configuration. The TNG package does include an on-line configuration function, but I could not get it to come up. The problems seemed to be file permissions to the TNG home directory I created for the package. After another week of messing with it, out of pure desperation, I renamed the TNG home directory to a name other than the recommended default name, and everything worked. I remain baffled; but I did give a small prayer of thanks to the Gods of the Internet for bestowing their favors upon me.

Once past that hurdle the installation went smooth. Right up until I moved to the TNG – WordPress integration. The integration includes a module that gets loaded into WordPress as a plugin. I performed the plugin installation, stepped through the configuration procedure, and excitedly pulled up the WordPress page that would display the integrated TNG homepage. The WordPress page was blank, completely free of any TNG software. Then I discovered the TNG plugin was causing my WordPress site to crash.

I spent another week on the TNG help forums hoping someone could provide clues to my integration problems. Although I received some helpful suggestions, nothing helped. As best as I can tell, the integration is dependent upon the particular WordPress theme selected, and I didn’t have the magic theme that would work. Not being willing to redo my theme, I regrouped and took a different track.

I’ve opted to go with running TNG as a separate sub-domain, appropriately configured as “genealogy.charest.net.” I’ve built some links in the TNG site back to this website, and built links here to connect to the TNG site. I actually think this two linked site approach might work better than trying for a full integration. I suspect as time goes on my genealogy site will take on it’s own “personality,” and will need to be developed separate from the blogs and photo albums on this site.

The TNG package doesn’t have the flexibility of themes the way WordPress does. It does offer a small selection of built-in themes, none of which I particularity like so I went with the least ugly choice. Strike one up for open-source vs. proprietary software. The TNG themes can be user-modified but that requires going into the configuration files and editing PHP scripts, something I don’t have the energy to do right now. I’m putting TNG customization on my To-Do list for when I’m really bored. Expect to not see any genealogy site appearance changes for few years…

Last week I started uploading new genealogy information.  All told, I have the makings of building a pretty nice genealogy site, given that I find the time to upload everything.

So please feel free to jump over to our new companion website “Our Family History: Genealogy of the Charest Family” and explore. The Charest family has an interesting family history, one that I’m very proud to be a part of.

He’s Got The Look

Programmer At Work

Programmer At Work

Regular readers of this humble website might have noticed that over the past few weeks we’ve gone through several changes in our look. Astute regular readers might have also noticed that on frequent occasions the look of this website was “Basic HTML,” a look where the actual layout was sucky basic lists of posts and navigation with no apparent effort made in layout.

There are reasons for all this.

First, the new clip art. The previous iteration of this website was built on software with limited ability to add images. It was possible, but frequently involved more effort than I was willing to put out. So, I only added photos when they really added to the story. With WordPress, adding images is extremely easy, not to mention that now there is a lot of open quality source clip art available for free. So some months back I decided to add at least one piece of clip art to my posts. As I had some free time, I went back and added clip art to most of my older posts, just because. I do think a nice piece of clip art can set the tone for the the story.

Which gets into the website look. The actual appearance of a website is based upon ”
themes”; a set of HTML files that define the fonts, title layout, headers and footers, and navigation system. In a well designed website the theme will be completely independent of the content and can be changed out relatively easy. WordPress has literally hundreds of different themes available for free, mostly well-written. The styles range from simple to extremely elaborate, and for-fee designs can be even more complex.

I opted for a “minimalist” style when I migrated to WordPress, and chose the theme “Chiron.” I customized it a bit and launched. I also added a set of additional plug-in software modules to increase functionality. Then I proceeded to add new content.

Over the next several months I noticed that on occasion, when I opened up my website, the theme layout was wrong. The layout looked like the theme had not loaded, and all I was seeing was basic HTML. I did some de-bugging, never found anything “wrong,” and after awhile things went back to normal again.

A couple of weeks ago I saw the problem appear again, and decided to change the theme  I was using. WordPress had gone though a number of major software updates over the previous several weeks and I noticed that my theme had never been updated, so I made an assumption that my theme was no longer compatible with the new WordPress software.

Like all good developers, I have a private testing site set-up to experiment on before going live. But, I’ve been a bit lazy these past months on tech stuff and my testing site was way out of date. Thinking that WordPress made changing themes as easy as changing socks I selected a new theme and went live.

Then I noticed that the new theme handled my summary and feature images differently than the previous theme. Summaries was the worse. I prefer to have only an opening paragraph on my home page with a link to the full story. I do this for two reasons;

  1. It keeps my home page cleaner, and provides a quick index to recent posts.
  2. I can get a better set of statistics on number of reads for my stories; not possible when the story is posted in full on the home page.

The first theme I used had an option for automatically adding a “Read More” link. The new theme I selected did not, so all my posts were full length on the home page. The only way to fix this would be to edit each and every post with a tag that manually broke the post with a “Read More” link. With over 200 posts, that I had just finished editing to add clip art (which also did not display as I originally intended) my choice was to try a different theme.

So over the next couple of weeks I changed themes every few days to see what might work better. I learned a lot, starting with each theme handles some function of content differently. Some themes actually “broke” layouts. After several tries I decided I needed to get my testing site back up to a mirror of the real site, and play there.

So I did, and tried out more themes. Meanwhile, my real site, with my original theme, was still not working. So today I decided to use a “minimalist” theme “Twenty Twelve,” a core WordPress theme that gets updated whenever core software changes. Hopefully, this will solve my issues with the theme periodically not working correctly.

The down-side is that this theme does not appear to support automatic summaries, so now I need to go back and edit all posts, again. This is the life of a Webmaster.


Tech Talk on Migrating Data Files

Programmer At Work

Programmer At Work

I’ve spent the past week on business travel in Southern Florida. As business trips go, this one wasn’t bad. The workload was relatively light and the weather here beat the hell out of weather in Washington, D.C. (for anyone who only wants to see ice in a drink, not on a road). The downside is that I’m not home, but Winnie always seem to take our temporary separations in stride.

The good part is that I had free time to migrate my on-line photo collection to this new website. It was a bit of a lift due to the number of photo images I had in my legacy site, and due to the differences in software.

WARNING: Tech Speak ahead! If technical talk causes you to roll your eyes and view your Smart Phone, stop reading now!

Ok, thanks for sticking around. As I was writing; Migrating photo images was a lift and I essentially ended up editing each file.

The first problem was the sheer number of photo images I’ve uploaded over the nine years I’ve been operating this website – 638. This actually accounts for only a small fraction of my total photo collection. With my legacy website I limited myself on photo uploads  as the software I used was pretty clunky and uploading photos took a lot of time and effort. Lucky me, now.

This leads to my second big problem of differences in software. The legacy photo software I used was “Gallery,” an open source package that in 2005 was pretty popular (and still is) as a stand-alone imaging package. The version I used, Version 1.0, had the advantage of integrating pretty nicely into the Content Management System (CMS) package I used named “PHP-Nuke.” PHP-Nuke was also pretty popular at the time. Time, however, marches on and my website didn’t.

I never upgraded my software once I got everything working right on the basis that I shouldn’t be fixing something that wasn’t broken. This approach worked well until last November when my website did break, and I discovered that I could no longer fix nine year old software.  PHP-Nuke had moved from my website version 7.6 to version 8.4, and when I upgraded everything not part of the PHP-Nuke core stopped working. This drove me to migrate to a new and supportable software package “WordPress,” which offers a lot of excellent plugins including a photo image package “NextGen.”

Once I learned how to use WordPress and migrated all my posts and feature stories I started working on my photo collection. Gallery v1.0 used flat files for storing image metadata and the NextGen loaded everything into a MySQL database (as it should).  I quickly learned that no software developers had ever taken the time to write a freely-available script to migrate Gallery v1.0 to NextGen, but I found a script to migrate Gallery v1.0 to the latest Gallery version v3.0 which did use a MySQL database. The data structure between Gallery and NextGen databases is different, but converting between different databases is pretty straightforward.

The script mostly worked, migrating metadata for about half of my images. There was no apparent rhyme or reason for migrating some images and not others, but I decided that half was better than none. With at least some metadata in a database I expected to be able to convert over to the new NextGen data format fairly easy.  This was still not to be.

I uploaded my images into the NextGen software and loaded the metadata I did have from Gallery. Then I discovered that the NextGen package could read all file metadata including file creation dates and camera data from electronic data, unlike Gallery v1.0 where I had to manually load metadata with each image I uploaded.  With my migration metadata just didn’t exist for many images and if there was no time/date stamp recognizable to the up-loader, NextGen assigned the timestamp of the upload event. I also discovered that titles and descriptions for most of images that did migrate over really sucked.

The end result was that I spent my free evenings this past week manually editing 638 photo images for titles, description, and tags.

As of now, I do have photographs once again available for viewing under the “Our Photography Albums” section of the sidebar menu with proper titles, descriptions, and tags. Timestamp data still has to be edited as required but I’m putting this off because it requires me to dig back into my files of 35mm negatives for dates. This might be a job for my next business trip…

I do hope our many viewers enjoy my photo collection. Since it will be so much easier to upload and post images with this new software I expect to upload a lot more images in the coming months! Next job will be to migrate genealogy files.

Hello World!

Hello World In Several Languages

Hello World In Several Languages

This is the first post in our new “Charest Family on the Web” website. We have a new look, and lots of new functionality, while eliminating the bugs and chunkiness of the old site. Also, hopefully, eliminate most of the security holes in the old site that seemingly encouraged hackers to engage about every six months with spammers constantly hitting.

This new website runs on “WordPress,” an Internet publishing system that seems to be one of the most popular open source website packages available. It’s optimized for writing and related content, appears to have a solid community of users and forum support, and a large set of plugins for expanded functionality.  One of the more important functions for me is the ability to quickly add new users and support moderated comment threads on open posts, features that with the former website were sorely lacking.

This new website is a long way from being fully operational. I’ve just finished importing and cleaning up all legacy stories. I still have the photo gallery and genealogy files to bring in. One thing that has surprised me is the scope of content that has accumulated over the past seven years of operation. I discovered that I had 195 stories and news posts, 638 different photo images, plus genealogy files that I have yet to quantify. Because the old software package is largely unsupported there are no ready-made utilities available to automatically convert data formats. I’ve had to work through the conversions while also learning how to use WordPress. Not impossible, but I won’t be spending much time playing on Face Book for a while!

I hope you’all enjoy the new look and (hopefully) user-friendly functions. Please feel free to request a user account and add your thoughts in the comments threads of future posts. I will be writing again, after a couple of dry years. All-in-all, this looks to be a great year ahead!

Choking Off the Conversation

Programmer At Work

Programmer At Work

…And other comments on spammers and hackers

When I started this website over two years ago, I envisioned a comfortable family-type website that could help our far-flung family keep in touch.  I wasn’t a newbie at running websites as I had operated a “commercial” website for the five years prior to my divorce (commercial in the sense that I had hopes of making money off it, not that I actually did make money).  I was well aware of the threats from hackers, and the annoyance of getting hundreds of daily spam e-mails offering everything from increased site traffic to earning thousands, even millions, of dollars overnight using whatever money-making system the particular spammer was pushing. It was about three years between the times I shut down my flagship “RCM TravelSite” to the time I started “The Charest Family on the Web.”  I have been shocked at how the Internet community has changed over those years.

My earlier website was largely a homegrown effort that ultimately used a hodgepodge of commercial, free, open source and in-house (I wrote it) software.  None of it worked together seamlessly and for the last year or so that I operated it, I was spending the bulk of my time trying to keep it together.  When I decided to start up this site, having a software package with the features I needed, that actually worked together, was important.

For the first year I used a light-weight package called “Cute Copy.”  It was easy to setup, and seemed to have all the features I wanted.  But after a year of use its limitations really showed, so I looked around for something to replace it.

I settled on PHP Nuke as my platform for a variety of reasons, not the least being that it was a pre-loaded package offered by my web-hosting company. I had used an earlier version on RCM TravelSite as my news section and had some working knowledge of it.  I also knew there was a LOT of third-party modules available that offered all the functionality I wanted.

So I set it up, experimented with site designs, and launched this noble experiment in high-tech family communications.  Over the next several months I located and installed add-ons including the Photo Gallery and Genealogy sections, and customized other modules to my personal preferences.

From the start, I wanted to build an on-line community.  I already knew the key to building a community was making it easy to communicate (I know, sounds simple).  So I setup this site to make it easy for visitors to make comments and add content.  Among other settings, I allowed any visitor to post comments even though I restricted news posts to registered members.

For the first several weeks, all was well.  Then I started paying attention to the comments showing up on my world-shattering news posts.  I was horrified.  Spammers had discovered this humble website and were using the comments section to post links to all types of trash.  I found ads for male performance drugs, male enhancement drugs, FDA non-approved diet drugs, in-line casinos, porn sites and God knows what else.  I was pissed.

So I restricted the comments to only registered members, thinking that having to first register might slow down spammers.  It did, for a few days.

Then I was hacked.  Some group who I believe call themselves the “Young Turks” hacked in and defaced my website, over-wrote several recent news posts and changed some of the database-driven formatting.  It took me the better part of a day of free time to undo the damage.  Then I spent almost a week’s worth of free time researching and installing PHP Nuke security patches.

I was still being plagued with spammers in the comments section, and now I was also dealing with spamm user’s accounts.  I couldn’t find a patch or figure out how to screen the comments, so finally I just blocked comments – or so I though.

While searching for security upgrades, I located several useful third-party add-ons including a guestbook module.  I had operated a guestbook on RCM TravelSite and enjoyed seeing the comments.  I knew there was a chance of a guestbook being abused by spammers, but on RCM TravelSite I’d had one spam message for at least every 10 valid ones.  So I tried a guestbook again here.

The guestbook had a screening function, so I thought it would be manageable.  I was quickly shocked at just how wrong I was.  Within days, I was receiving several daily spam messages.  After several weeks, I was receiving up to twelve messages daily, and when I started getting porn photos posted, I had enough.  I locked out the guestbook until such time as I could come up with a more effective screening.

By this time I had activated the weblinks module.  Again, within days of activating this module I started getting links submissions for websites including porn, on-line drug sellers, casinos, and insurance companies.  I’ve been able to filter the trash out, but it’s one more irritant I’m dealing with.

Meanwhile, I was getting several dozen daily spam e-mails on the webmaster e-mail account I had to delete.

Last Friday I made my daily website check and discovered I had been hacked again, with my website defaced as before.  The same group as previous appears to be the latest culprit.  It seems my security upgrades had done some good, as they didn’t do as much damage as before.  Or maybe they just didn’t bother, I really don’t know.  But it took a day’s worth of free time to undo the damage.

In the process of undoing this latest hack, I discovered spammers had gotten into the comments section I though had been deactivated, I found dozens of spam posts on archived news stories.  So I spent more time deleting the trash, deleting the spam users who had taken the time to create user accounts just to post their spam, and relocking the comments section.

So this is where my noble experiment in family-forum communications has come to:

  • No Comments allowed
  • No Active Guestbook
  • No one allowed posting News Stories except administrator
  • No one allowed commenting on the photos except administrator
  • I pretty much ignore all webmaster e-mails.  There’s so much spam, I’d probably miss seeing any real e-mail
  • I don’t dare activate the Bulletin Board Forums, to avoid one more source of spammers
  • No one can submit genealogy info other than through an e-mail, which I probably wouldn’t notice due to all the spam
  • I’m resigned to being hacked on a regular basis, and can only hope the SOBs don’t spend too much time mucking around when they do hack me

So you’all need to excuse me for feeling irritated by spammers and hackers now.  I’d call these people slime, except actual slime mold can’t help being what it is, and these people could. Calling them slime would be libelous to the mold-type slime.

As far as hackers go, I need to tell these cretins to go get a life.  Hacking the CIA’s website might give them bragging rights. Hacking into Bank of America’s website might gain them lots of lucrative account numbers.  Hacking into a family website operating on free software with a nominal dozen daily hits gains them…?

For the legitimate visitors to this humble website who’d like to actually participate in a conversation, I’m sorry.  I don’t have the programming skills, or the time to learn, to build a spam-proof hack-proof website.  Until spammers and hackers all crawl back under the rocks they came from, I’m going to have to keep tight security restrictions on.  If you really want to contribute something worthwhile, you may contact me using private messaging (that you can access that by creating a user account) and let me know.

I’ll be happy to create a user account that allows someone else to post worthwhile news stories or other content.