Javascript is a core part of web experience today.  In fact, I’d say that, on the desktop in the full browser front, if your browser doesn’t support at least HTML 4, javascript, and CSS 2, you’re not playing with the right tools.  After all, we expect that people can parse HTML, why not expect that javascript is a pre-requisite for web usage?

Some of us go to great pains to make sure our sites work should a user have javascript disabled.  But I’m actually considering the opposite: hiding certain critical elements if you don’t have javascript enabled to ensure that each visitor is on an even playing field.  Wrapping submit buttons in jQuery’s append() method, submitting data on click(), and plentifully exchanging JSON data via AJAX throughout ought to properly cripple participation of those who opt out of script execution on my site.

It all comes down to this: if you want your site to reach the widest audience possible, you need to anticipate that the client may not allow you scripting capability.  Conversely, on our intranet, and maybe one day on my websites, I’m doing the opposite: if you want to use the site, you’ve got to enable javascript: if you don’t, well… your loss.

     $('a[rel*=fancybox]').fancybox({
          'frameWidth' : 500,
          'frameHeight' : 465,
          'hideOnContentClick' : false,
          'centerOnScroll' : true,
     });

This is the fix:

     $('a[rel*=fancybox]').fancybox({
          'frameWidth' : 500,
          'frameHeight' : 465,
          'hideOnContentClick' : false,
          'centerOnScroll' : true
     });

See the difference? Yeah, neither did I. The difference is the last comma in the argument list.

That’s 3 consecutive major versions of IE that have been absolutely crap.  Why anyone continues to use IE is beyond me.  IE: sucking hard since version 5.

First of all, extensions such as Stylish and Firebug are invaluable.  In fact, scratch Firebug, the default Firefox error console alone is aces to me.

Is there anyone who can tell me why no browser besides Firefox has a “View Background Image” link even as an option? How come no other browser has developer friendly stuff? I know that the Web Inspector in Webkit browsers is really cool – I love Webkit – but ultimately, it’s Firefox I often resort to when I’m doing real work.

The interesting thing here is that I’ve browsed the source of Slash, Scoop, phpNuke, and now WordPress, and all of them are definitively more complex and much heavier than the entire OSNews codebase. Now, before you jump all over me – firstly, Slash and Scoop are Perl, and I don’t really read Perl, so I can’t speak as an expert there.  Secondly, WordPress and Nuke both are very portable and dynamic, whereas OSNews has a narrow focus and, location-wise, is very static.  But that aside, OSNews has withstood simultaneous link bombs from Slashdot and Digg.  As amazing as WordPress is, it’s mostly amazing that it functions at all and loads in less than 2 minutes per page with as much going on as I can see behind the scenes.   That’s not a cut on WordPress, by the way.

In fact, if anything , what is really impressed upon me is how smooth and simple OSNews code is, if I may be so bold.  OSNews runs superfast due, in part, to lots of creative caching, some on-demand, some via cron.  But it also does so because of highly efficient queries that are measured for speed on their JOINs, meaning in some cases, it’s faster to do 20 simple queries than one complex one, or build a long and scary chain of “OR x=a OR x=b OR x=c OR x=d…”  Watching WordPress code in action is really fun for me, but watching OSNews work knowing what I now know about how much work PHP can cram into its threads is even more fun.

At long last, the iPhone will become what it was destined to be. In June, when the iPhone 2.0 update is released, the iPhone’s true potential will be unlocked. VoIP? Sure, why not!? Games? You betcha. Exchange, ActiveSync, Remote Wipe, 802.1X? Check. How about access to the entire SDK via XCode, a compact framework (Cocoa Touch), a native emulator, and access to the SQLite databases present in the iPhone file system? Yup. Lastly, how about the most innovative platform in the last 20 years that has single handedly made the mobile web viable? Present and accounted for.

In fact, the iPhone is a new generation, and it’s been grunting along the sidelines as a gloried browser. But come iPhone 2.0, it will validate itself as one of the most amazing devices out there.

The fact is, it’s rarely PHP that crashes a website. It’s usually overloading MySQL and/or bandwidth limitations. Obviously, we can’t do much about bandwidth. If you’re on a really cool host, they may be psyched for your Digging or Slashdotting, but most are not particularly keen on you using too much processor, too much bandwidth, or too many MySQL connections. Although MySQL is capable of amazing scaling, rarely do web hosts leverage it properly, and even if they did, rarely do most programmers know how to read into slow queries, find inefficient code like loops within loops, and debug a site performance before it’s too late.

The site linked from Digg shares a very lame trick, let’s examine:

//put this code at the very beginning of your page, before anything else
$randomdigg = rand(1,4);
if($randomdigg != 1) {
exit("Thanks for visiting, but the site is under a great deal of 
stress due to being on the front page of Digg. 
Please try back in a few moments. Thanks!");
}

Ok, so essentially, approximately 25% of the time, the page will work, the other 75% of the time, the user will get the message and be asked to refresh later. That’s really lame. First off, unless you want your website to be functional only 25% of the time, you have to be there babysitting it when it hits front page. Also, you need to remove this code. But what if you wanted everything to happen automagically? What if your site could withstand a digging, a slashdotting, a redditing, an OSNews’ing, a Farking, whatever…. all at once, without being forewarned?

Although there are lots of [[http://www.danga.com/memcached/|tools that can do magic for you]], you can leverage plain old PHP to do this dirty work for you, and it’s easy to integrate into your site.

Here’s the premise, we examine the referrer of the page, and if the page is from one or many domains that you specify, you build a cache ON REQUEST. Then, you serve the cached version. When the cache stales, say after minute or two, you refresh it, as needed, again, on request. And the best part is, it’s easy to do.

So, let’s review the code. First, at the top of the page, let’s do this:

$rfr = $_SERVER['HTTP_REFERER']; //get the referring site
$op = '';
if(strstr($rfr,'digg.com')) { $op="do-cache"; } 
$cache= "/path/to/mysite/cache/".$postid.".html";

Let’s take it line by line. First, we get the referring site. If it is digg.com, we’re going to turn the cache on by setting $op to “do-cache.” Note that we preset the $op variable to null to prevent abuse. Next, we define the file we’re going to use as the cache. Since we might want to do this for any entry/page, we’ll give it a unique address. Remember that you do not have to use .html as an extension, since it’s static code anyway. I’ve seen “id.cache”, “id.txt”, and even just “id”. It doesn’t matter, it will be included in a PHP page and spit out as X/HTML anyway.

if(file_exists($cache)) {
	$fmt = filemtime($cache); 
	if($fmt > time()-180) { 
		header("Location: http://mysite.com/cache/".$id.".html"); 
		exit(); 
	} elseif($fmt > time()-1800) { 
         //cache will reload if it's been reloaded 
         //in the last half hour
	   $op=="do-cache"; 
	} else {
	   $op='';
	}
}

This might seem overwhelming, but it’s really not. Check it out. If the cache exists, regardless of referrer, we’re going to check it. The filemtime() function will tell us when the cache was last refreshed. If it reports a time in the last 180 seconds (you can change that number), it will redirect you to the cache. PHP will barely have to do any work, Apache will barely have to do any work, and MySQL will not be touched at all. Almost any host can hang in there to serve static traffic like this.

But what about if the cache is old? Once your site is no longer getting hits every three minutes, for the next 27 minutes, it will blindly reload the cache. It will keep the cache going as long as the page has been hit in the last 30 minutes. After no one hits the cache for 30 minutes, it will go back to fully dynamic. When someone hits the page from digg again, it will start the caching process again. In this case, we’re going to let the cache rebuild. Enter the next part of the code.

if($op=="do-cache") {
	ob_clean(); 
	ob_start(); 
}

If the $op variable is set to “do-cache”, our mission is… uh… to build or rebuild the cache. So, we begin with two simple PHP standard functions, ob_clean() and ob_start(). The “ob” in these functions stands for “output buffering.” When you buffer your output, nothing is echoed by the script, but rather written to a buffer. In short, it’s all stored up in memory for eventual dumping later. Now we stop the caching scripts and just write our normal HTML via PHP and MySQL.

At the end of the page – the very end, after your html element is closed, you’ll need this last bit.

if($op=="do-cache") { 
	$buffer = ob_get_contents(); 
	$fp = @fopen($cache,'w'); 
	if($fp) { 
		fwrite($fp,$buffer); 
	}
	@fclose($fp); 
	ob_get_flush();  	
}

What’s up with this? Again, if we’re building the cache, we’re getting the contents of the buffer, which contains all of our page. Then we’re opening the cache file we specified above, and lastly, dumping the contents of the buffer into the cache file. Our final action is to clear the buffer, which is unnecessary to us now anyway.

That’s it. Your cache is reloaded, and will be served for the next three minutes, after which it will reload the cache every 3 minutes until no one hits the site for 30 minutes. Then it will go back to dynamic. If another digg visitor comes, it will recache and serve the cache. Having written several types of caching mechanisms for large sites myself, I can tell you this “cache on demand” method works. On OSNews v4, we use this for things like user-specific RSS files, which aren’t plausibly built via cron jobs and, frankly, we don’t want to allow access to MySQL without limit. They reload every 60 minutes, regardless of referer.

If you maintain a site and have any concern about being Dugg, I encourage you to research some sort of caching mechanism. Your web host will thank you, and you will be thankful for the maintained uptime and high hit count.

So, I got a phone call from the director of our customer contact department this week. He was concerned about the reports. He made a decision last week that when a call came in that was a lead – in other words, a non-customer, that his people would fill the phone number from the caller ID into the account number field. But when he ran his export reports, he found that hisn techs had entered this phone number for ALL of the calls: 429-496-7295. That’s weird, he said. So he called me and asked why that was. I checked all the calls and most were from one woman, so my first instinct was “Check if her browser has autocomplete turned on”. But he swore that he tried it too and gotten the same result.

I checked the database and sure enough, it was right there: 429-496-7295, in all of the fields. So I went back to the code. In short, I took the input from the form, and declared it like this:

$accountnum = (int) $_POST['accountnum'];

Pretty straightforward: explicitly declare the type. So, I started my debugging by attempting to manually enter the data into the database. Sure enough: the account key field showed this: 4294967295.

So, I went back to the PHP and started by dumping out the raw SQL query:

INSERT INTO calls ('','x','x','x','4294967295','x','x');

What? So the database automatically converts it to this weird phone number and PHP does too? Suddenly it occured to me. One of the benefits of 64-bit computing is the ability to address more memory. There are limits to what can be done in 32-bit computing, and one is that integers have a limit! In this case, a database field called “integer” is limited to numbers between -2,147,483,648 and +2,147,483,647. It just so happens that the number is the same length as a US phone number – 10 digits. Changing the db field to “BIGINT” allowed me to manually run the SQL query and it worked. But the app still didn’t.

PHP’s int() and (int) $var syntaxes both conform to the integer limit. So I devised a work around:

$ac = $_POST['accountnum'];
if(!is_numeric($ac) { $ac = (int) $ac; }

It’s not gorgeous, but it will more than suffice for an internal app. We web programmers don’t usually have to deal with big integers, so it’s totally possible that web developers would never have had occasion to handle a situation like this. Here’s looking forward to native 64-bit for our next server, though.

I love Camino, really I do. But recently, its limitations have been bothering me. I prefer my tabs in a very specific order and often I have several tabs open. If ever I close a tab by mistake, I cannot get that same order without doing tons of work or re-launching. Safari 3 draggable tabs.

One of the things that used to bother me about Safari was that there was no “New Tab” button available for the toolbar. There is now. It’s also got great keychain integration, private browing, the original embedded RSS, true Aqua widgets, resizable text boxes, easy PDF integration, and it’s super-fast.

Camino doesn’t support Ad-Block, but rather, stylesheet-based filtering. Safari does that too, by default, and it’s even easier to use than it is in Camino. Safari doesn’t have any Flash problems and once you add “Safari Stand” and enable the debug menu, you have a perfect drop in replacement.

My biggest complaint about Camino was the lack of development tools. It doesn’t have a Javascript debugger (ChimericalConsole never worked me for), doesn’t have a decent source viewer, doesn’t have many third party hacks to add functionality – it’s a browser for users, not developers. Without XUL, it’s tough to add features easily. And that made it tough to use for me. When I did any serious work, I’d always switch to Opera or, more recently, Safari 3. Safari 3′s Inspector is just awesome.

So… for now, I am Opera on Windows and Safari on Mac. My browser requirements are more demanding than most. I have felt for some time that Opera and Firefox on Mac just “feel” wrong, they don’t fit. So we’ll see how the Safari experiment goes.

If you want to make a mirror of your blogsome blog, you can use a single very powerful command to generate a snapshot of it from any Linux machine or Windows with Cygwin installed:

wget -k -m -r http://url

But this will only create a static HTML mirror of your website. It won’t allow you manipulate content or put it into another database. That leaves only one way to do it – request the entire site page by page, and parse each page individually. RSS is not reliable here, as most people have it set to only 15 or so items and parsing an enormous page make make PHP or your server run out of memory or alloted script execution time.

It’s a multi-step process, to be certain. It was actually painful to go through the process. Requesting over and over, debugging the script line by line. It takes several steps to get things right. But, eventually, I did it. I was able to export a Blogsome blog in its entirety – every entry, all the categories, all the comments with emails and websites …everything.

Slashdot was the same. It took some tinkering, but eventually, I was able to backup a very lengthy Slashdot journal. Again, in its entirety. I got every post, the date, etc. It was not simple, but it worked well. And I not only got them backed up, I merged them into a single database serving up… Small Axe 0.6 (which was a whole adventure of its own, taking current firsttube.com code and “neutralizing” it). Suffice it to say, when I saw all entries working and served in Small Axe, I had a huge smile. Turns out that the person I was doing this for decided against Small Axe (only because it doesn’t yet offer all the bells and whistles WordPress does, even if it is a beast). But it was irrelevant – the hard part was getting the data properly, and that’s done. Migrating to *any* blog database is possible if you have the time, inclination, and skill to write a SQL export/import script.

Here’s how it works:

* Cycle through each page of the blogsome blog. On each page, we get the entries, the URL, the postid, and other relevant info. We set a flag on each item to 0.
* As we retreive the items, we correct the path to images, spacing, smilies, etc.
* Then we cycle through each page individually. We have all the URLs already, so we go through each one and parse comments. It’s important to know that comment owner comments are marked up differently. As we get the comments, we upodate the flag and let the script run on its own. Our 900+ items took about an hour by meta-refreshing the fully rendered page every 3 seconds.
* As we go through the comments, we strip tags we don’t want, we fix emoticons, we fix internal links, spacing, etc. We must expose emails temporarily if we want them to transfer over.
* Finally, we import them all into a central database with an agreed schema.

If you have a Blogsome blog and/or a Slashdot journal you need backed up, I can help you do it. It’s not a simple process, but it is very accurate and preserves whatever data is exposed via HTML. So for the right barter, I would be very motivated to help. If I can simplify the process, I may create an open script to do this. But for now, I’ve got the code.

On a related note, I’ll probably release an updated version of Small Axe sometime in the not too distant future, because the amount of changing I’ve done and all the features I’ve just implemented are killer. Small Axe is FAR from WordPress caliber tested, but it’s SUPER simple and can do all the basics of a normal blog, including templating, smart per-domain caching, blocking by ip, username, email, or keyword, gravatar support, tons of configuration options, RSS and Atom support, threaded commenting, post locking, post expiring, browser identification, slug-based permalinking, and much more.

The new site is much simpler, easier to use, and less overwhelming. I will be adding some of the previous material – links and such – back in over the next few weeks.

I’ve also discarded the concept of “collapsed threading” vs. “expanded threading.” All threads will be expanded, because it’s less code and it’s easier to maintain. Building on that, comments below your “threshold” will not require a reload, but rather just an “expand” to be viewed. Lots of things, like viewing below-threshold comments, moderation, etc will not require a page load. Only some of this requires actual AJAX, the rest is just Javascript.

I’ve also cleaned up the URIs quite a bit. First off, pages no longer have a file extension. If we one day rewrite the site in Python, or Ruby on Rails, or whatever, we’ll be future compatible. I will be redircting all working pages to the proper page in the new site, so links will still work. It will be nice to reference my comments at www.osnews.com/user/ascheinberg/comments, my bookmarks at www.osnews.com/user/ascheinberg/bookmarks, my “starred stories” (more on that below) at www.osnews.com/user/ascheinberg/starred, my friends and fans at www.osnews.com/user/ascheinberg/relationships, and my public profile at just www.osnews.com/user/ascheinberg.

That said, several of the pages that exist now will go bye-bye in favor of better organized pages. Besides just the user info pages, you’ll find that thread.php, subthread.php, permalink.php, read_thread.php all pretty much capture the same information. Also, reply.php will be blended with the regular comment posting section. So a code cleanup is in order.

I’m rewriting in XHTML, although I’m not worrying about validation just yet. That said, it’s almost entirely styled with CSS, and thus far, there are no tables. That’s not to say there will not be several tables, but for now, layout is solely CSS.

I came up with the concept of “starred stories” because I want to make “recommendations” more of a suggestion for other readers. I will be expanding on this point in future posts, but trust me, it will make sense or it will be left out.

I also am currently supporting something I’ve not decided on: avatars. It’s not necessary, but since I wrote an image resizing script for the admin backend for automatic thumbnailing of screenshots, I can easily support an automatic resize for avatars, and honestly, they look nice.

The threading implementation, believe it or not, is actually faster than the current implementation because it’s much sleeker code. I’s very cool to quickly switch between thresholds and view modes without setting cookies or updating profiles.

One of the very cool features I’ve already got working is pretty much a tagging system, except instead of liberal use of tags, it will be used to “group” stories. Let’s say it’s time for WinHEC – we can group all stories from there as “WinHEC 2007″ or something. Or “CES.” Or “Macworld,” “LinuxExpo,” or even “Fedora Core 7.” It remains to be seen HOW we will use this system, but it’s already in the actual live database. Note that it will NOT show up in the current version of OSNews (v3).

Lastly, for today’s OSN4 details, I have one more fun feature I’ve toyed with for some time: user defined conversations. Essentially, it allows users to create their own OSNews article, or even just a blurb for discussion. I don’t know the specifics, this isn’t coded yet, but I think this could be really neat – let’s say you start a conversation: “Predict the specs for the next macbooks” or “Your experiences with Windows Vista.” Could be fun, right? We’ll see whether or not this turns out to make the cut.

Lastly, I want to share one more thing: RSS everywhere. Every relevant page will have an RSS feed. I’m excited about this.

So far, I have a working site that displays the main pages (the index, the story page, and the comment pages, with a working working threading implementation). More as it comes.

As always, all of this OSN4 stuff is subject to change.

In the meantime, this new project is hopefully going to be launched in the next week or two and I’m hoping it makes a splash. In the meantime, I need a few beta testers to provide some feedback. If you’d like to help test a new site we’ve been working on, please post a comment below and I can explain what to do.

Now – as of right now, you can only reply once, like digg, but it supports two deep. This is only because I haven’t found a more elegant way to allow deeper threading in the code. As soon as I do, I will be allowing deeper threading, probably 3 or 4 comments deep for aesthetic reasons.

The entire thing is done in CSS, unlike OSNews which is hard coded to the pixel with very complex code that really makes me uncomfortable when I see it. This is much nicer. It’s all done via templates, and there are only two – one for comments and one for threads, and this will probably change to one soon too.

- Small Axe Encryption (which is VERY breakable (being as though the decrypt code is actually available, but a new version, which uses a password to decode, will eventually be out).

- Small Axe PHP XML-RPC API. YIKES. What a mouthful. It’s by far the easiest API out there. I have searched high and low for an API. Version 0.1 is so simple it’s crazy.

- Dango 2.1. A simple discussion board. Version 3 will be a complete rewrite, so this is a nice mini-step.