So, as I mentioned before, Flot is an awesome jQuery plugin for making on-the-fly graphs. The twitter api supports a JSONP mode for its output. 100 lines of javascipt (well, 60 if you condense whitespace and braces, commas… less if the goal was to write small code).
What I don’t understand about what I’ve done here is JSONP, which I first read about here, at remy sharp’s blog… Why doesn’t it get more press. Solves the problem that I would mentally run in to all the time when considering various mash-ups and takes on various internet APIs, that of cross-domain limitations with XMLHTTPRequest.
The heart of the magic is demonstrated in this incredibly simple example:
$.getJSON('http://twitter.com/statuses/user_timeline.json?id=CDMoyer&callback=?',
twitterCallback);function twitterCallback(timeline) {
alert(’Twitter Username: ’ + timeline0.user.screen_name);
}
The magic happens in two places. First, jQuery replaces the ? in callback=? piece of the URL with a function name that references to the callback you pass to getJSON(). Second, the twitter API takes the callback parameter and wraps the JSON result structure in a function call with that name. The end result is that the JSON structure is passed to a function in the namespace of your document, circumventing the same-domain origin policy
So far Scoble has reached the highest TPH (8.0) I’ve seen, looking at various popular tweeters.
I read about Flot a couple months back and have wanted an excuse to play with it, as it looked pretty slick. Unfortunately, not actual opportunity presented itself, so I had to just make something up. The neat thing is how simple it is to make a basic graph, but how you can achieve some awesome effects through various options. The latter examples on the Flot site show of some great features.
var data = [[0,0], [1,1], [4,5], [123,2], [12.3]];
$.plot($('#graph_area'), [data]);And that’s it. You need to make sure and have dimensions defined for the target element, and Flot will automatically scale your graph based on your data. If you want more data sets, you pass in more arrays inside the secord argument. plot() also takes a third argument, options which lets you controll the appearance and more advanced features.
Inspired by Oliver Steele’s Sequentially Library and this post by Felix Geisendörfer, I created a very simple jQuery plugin.
It provides a drop-in replacement for each(), which performs the loop in iterations and pauses between each iteration. You’d only do this if you are iterating over a largish set and performing non-trivial operations. Basically, if the browser freezes, this is a decent solution as the call to setTimeout() allows the browsers UI thread to get a slice of the action and prevents spinning cursor syndrome.
If you wrote:
$('foo').each(function () { alert($(this).html()); });
$('foo').throttle(function () { alert($(this).html()); });Or:
$('foo').throttle(
function () { alert($(this).html()); },
{per: 100, pause: 10});
MikeOS
MikeOS
If you like to read code, this is the OS for you. It’s a simple 16-bit OS written with the goal of demonstrating the basics of OS development. I’m not much of an assembly programmer (at all), but I was able to grab the tarball and read through, and learn a few tidbits right off the bat. Oh, and it’s got tetris. ;)
Jones Forth
Similar to MikeOS, Jones Forth is a wonderfully literate implementation of a FORTH compiler. You can just jump right to the source file and start learning. To quote the author:
FORTH is one of those alien languages which most working programmers regard in the same way as Haskell, LISP, and so on. Something so strange that they’d rather any thoughts of it just go away so they can get on with writing this paying code. But that’s wrong and if you care at all about programming then you should at least understand all these languages, even if you will never use them.
Three Small Javascript Libraries
A good post by Oliver Steele, describing three interesting libraries that really flex a bit of javscript, beyond the everyday DOM manipulation. Fluently – which allows you to easily create “chainable methods” (ala jQuery), MOP JS – which allows for some slick metaprogramming around asynchronous calls, and Collections JS – implements some nice utilities for collections.
TaffyDB – A Javascript DB
At first, my reaction to this was ,“Why?!” But then I mulled it over a bit more. I can see quite a few uses for this, especially in this modern era of “never reload the page.” I’m envisioning stateful browser apps neatly keeping their state, and transacting with the servide data store through tidy little JSON snippets. [via Joe’s Blog]
Normally, when defining a large array in PHP, I’ll write it like so:
$foo = array(
'some_key' => $val,
'other_key' => $val2,
'also_this' => $val3,
);Note the comma on the ‘also_this’ line. I’ve gotten into this habit, as it makes it much simpler to later rearrange the order of these elements or add another. As I’ve been playing with jQuery, I’ve had cause to create assorted JSON data structures, and wrote a bit of code like so:
$('#slider').slider(
{
handle: '.handle',
minValue: 0,
maxValue: 9000,
steps: 9,
stop: slider_slid_callback,
}
);(See the trailing comma?) Well, this works great… in FireFox. In IE?
Line 13
Error: expected identifier, string or number.
Normally I’d complain about IE here, but looking at Introducing Json I see this defintion for a JSON object (doh!):
object
{}
{ members }
members
pair
pair , members
In a startling development (to me), my small blog started getting spammed like crazy. Out of 100+ posts, a handful were legitimate. After going through the process of manually deleting them all, I decided to invest a bit of time in a CAPTCHA system for my blosxom blogging software. Having seen the reCAPTCHA interface on several sites, and read about it before, I decided to start there.
Basically, reCAPTCHA provides some incredibly simple APIs to add a CAPTCHA to your site. The neat part about it, though, is that the images presented to the user are actually difficult to scan words from real-life books. The results of people submitting these CAPTCHAs are combined to improve the digitization of hard-copy archives. The learn more page on the reCAPTCHA site has a more detailed explanation.
Step 1 – Sign Up, Get Keys
Step 2 – Add the the interface to the front end
<script type="text/javascript"
src="http://api.recaptcha.net/challenge?k=$MYKEY">
</script>This javascript (and iframe) API takes several parameters, allowing you to customize the look and feel of the interface.
Step 3 – Validate the submitted reCAPTCHA
As bloxsom is written in perl, I went with the CPAN module Captcha::reCAPTCHA.
use Captcha::reCAPTCHA;
my $rec = Captcha::reCAPTCHA->new;my $rec_res = $rec→check_answer(‘6LdWHQEAAAAAAORoD7pXt_BAHRdsZ2GPjmcFdWWi’, $ENV{’REMOTE_ADDR’},
param(‘recaptcha_challenge_field’), param(‘recaptcha_response_field’));
// [snip some lines]
if (!$good || !$rec_res→{is_valid}) {
$comment_response = “Some required fields were missing.”;
if (!$rec_res→{is_valid}) {
$comment_response = ‘Please verify that you are human, with the reCAPTCHA!’;
}
}
Obviously, not the most elegant code, but it was an incredibly simple process… and my blog has been spam free for 3 weeks!
jQuery rocks. After playing with YUI, EXT, and Mootools, and using Prototype/Scriptaculous for over a year, I am now completely sold on jQuery. It was a a post on err.the_blog, which first got me to take a look.
The Basics
As an example of why I am so enamored with jQuery, this simple javascript calendar required so little javascript, for so much functionality, that it blew me away. The whole idea is that you can chain various methods onto the results of simple selectors. Methods for animation, manipulation, and bindings are all provided. A snippet from my calendar example:
jQuery(function () { // jQuery() sets a callback for when the page is ready to go
$('#calendar .row .day').hover(
function() { $(this).addClass("active_day"); },
function() { $(this).removeClass("active_day"); }
)
}So, a simple CSS selector, and a utility function that automatically binds a function to the mouseover and mouseout events of each matched element. addClass() and removeClass() make it incredibly simple to provide visual user feedback. The simplicity of traversing the DOM in this manner, with #ids, .classes, and tag types, is incredible. Enhanced selectors such as :checked and :text are included to make finding the right element easy.
Simple Animation
As another example, the calendar includes a hidden div positioned at the top of the window. When a message needs to be shown to the user, this function takes care of showing the div, animating it’s appearance, updating the textual message, rounding it’s corners, and then hiding it again:
function show_results(str) {
$("#results").html(str);
$("#results").corner("bottom");
$("#results").slideDown();
setTimeout(function() {$("#results").slideUp("slow");}, 4000);
}Unobtrusive
I think what makes jQuery stand out to me, is how it really lends itself to the unobtrusive javascript paradigm. The better you semantically mark up your page, the more easily jQuery is to add to the page.
Recently, I’ve wanted to become aquainted with Git, the open source Version Control System that Linus Torvalds wrote after moving away from BitKeeper. My past SCM experience has been with CVS and Subversion, and the Git model varies significantly. The main difference being that Git is entirely distributed… Essentially, every user has a complete copy of the repository including history and metadata. They work with other repostiories by accepting patches.
If you’ve worked long enough with source code, you’re familiar with patches, probably enough that the network of repostitories connected and synced up by a series of patches described above scares you. It sounds a lot harrier than it ends up being in practice.
For my examples, I’ll walk through the processes I’ve used while working with Io, the neat little language that got me using Git in the first place. Keep in mind, I’m a relative Git newbie and my vantage is that get moving quickly with projects using Git.
git-core package originally, and it was missing some incredibly useful features, git rebase -i being the most important.> git clone git://www.iolanguage.com/Io
Initialized empty Git repository in /home/cdmoyer/foo/Io/.git/
remote: Generating pack...
remote: Done counting 7812 objects.
remote: Deltifying 7812 objects.
remote: 100% (7812/7812) done
Indexing 7812 objects...
remote: Total 7812 (delta 3799), reused 7468 (delta 3566)
100% (7812/7812) done
Resolving 3799 deltas...
100% (3799/3799) done> git log
...
> git log Makefile
...
> git status
# On branch master
nothing to commit (working directory clean)steve (it’s already named origin, but when you start to work with multiple repositories, a mnemonic name is helpful). We’ll also use git fetch to ensure that our local copy of steve is still up to date:> git remote add steve git://www.iolanguage.com/Io
get fetch stevesteve/master and master, which is our local copy. We haven’t changed anything, so we won’t see any changes. master is the name automatically given to the branch you checked out and started with. I’ve found it’s best to leave this branch alone, letting it represent the state of steve/master, and use it for merging changes in from other sources. I’ll periodically update master from steve, this is done in one of two ways:> git pull steveor
> get fetch steve
git merge steve/masterThey seem to be synonymous.
At this point, if you’re a CVS or Subversion user, you’re a bit concerned… branches are generally no fun. But under Git, they are pretty manageable. Next we’ll walk through creating a branch, doing work on it, and then creating a patch.
> git branch foo
> git branch
foo
* master> git checkout foo
Switched to branch "foo"
> git branch
* foo
master> echo "# a comment" >> Makefile
> git status
# On branch foo
# Changed but not updated:
# (use "git add <file>..." to update what will be committed)
#
# modified: Makefile
#
no changes added to commit (use "git add" and/or "git commit -a")> git add Makefile
> git status
# On branch foo
# Changes to be committed:
# (use "git reset HEAD <file>..." to unstage)
#
# modified: Makefile
#> git commit -m"Modified the makefile" Makefile
Created commit 14ff3a4: Modified the makefile
1 files changed, 1 insertions(+), 0 deletions(-)
> git status
# On branch foo
nothing to commit (working directory clean)Note: Locally modified and added files follow you when you change branches, so be careful with files that you modify and leave uncommitted as you move around.
> git checkout master> git pull steve> git checkout foomaster, and then apply your changes on top. If there is a conflict, it will drop you to the command-line for fixing any issues.> git rebase -i master
... Uh oh, we had a conflict!
> vi Makefile
> git add Makefile
> git commit Makefile
> git rebase --continueIf we couldn’t fix the merge conflicts, we could do git rebase --abort
> git checkout foo> git rebase -i master
# generally change all but the first "edit" into "squash" ... you'll see what I mean.> git diff --color master> git format-patch master
0001-Modified-the-makefile.patch> git config user.name Chris Moyer
> git config user.email chris@inarow.netgit diff will only show you changes since your last addition, to see since your last commit:> git diff --cached> git grep "error"> git branch -D branchnameHopefully this will give you enough to get started. It’s server me well so far. As I come across other common, useful or complex operations, I’ll write them up. Good Luck!
OSWD: Two Designs
Open Source Web Design
If you’re mainly a programmer, but would love to turn out sites with a bit more fit and finish, this site is for you. They provide packaged “site designs” which include sample HTML, css and images in a ready-to-download zip file. Lots of styles to choose from, and frequent new additions make it very helpful. [via lifehacker]
MySQL Performance Blog
This is one of those sites/blogs that you find while researching one topic, and then just keep reading and reading. Tons of well written, informative and helpful entries. Great discussion in the comments and on the attached forums.
The post that first sucked me in was Why MySQL could be slow with large tables ?
Benchmarking [php] magic
A great entry that considers the cost of many of the new features in the PHP5 object model. Also situated on another great blog that’s worth adding to your feed collection.
Amazon SimpleDB
A very interesting addition to their much talked about Web Service suite, SimpleDB provides a database (or directory service) in their computing cloud, along side S3, EC2 and others. With a pay-as-you-go model and some interesting features, this is the kind of environment that is sure to foster some awesome new projects.
Brady Forrest provides a great overview of the product and pricing on O’Reilly Radar.
Don’t let the lights go out on your site.
At some point, most successful LAMP applications will reach a critical mass of users, usage, data or other milestone, and a previously smooth running application will grind to a halt. Sometimes the degradation comes in waves, a blessing and a curse — it’s harder to troubleshoot periodic slow-downs, but it will affect fewer of your users. In either case, the first step to solving your problem is generally the same… Better logging. While analysis of the code and system may be helpful, the real bottlenecks are rarely obvious. (Or you wouldn’t have written the code that way, to begin with.)
So, to the point: you’re in this situation, the app is in production, what steps can you take to find the bottlenecks?
my.cnf (often /etc/mysql/my.cnf), and ensure that you have settings like this:set-variable=long_query_time=1
log-slow-queries=/var/log/mysql/slow_query.log
/var/log/mysql/mysql.err) for any errors along the lines of “Could not use /var/log/slow_query.log for logging (error 13)”_. If you see these, create the @slowquery.log@ and set it’s ownership to that of your mysql user.slow_query.log will begin to accumulate queries. The actual format of the slow log is bit verbose, but mysqldumpslow, a perl script included with most mysql installations can parse it and produce some more meaningful output. It will take various integers in your queries (a user_id, thread_id, etc) and generalize them, so you can locate types, instead of specific queries.> mysqldumpslow -t=10 -s=t /var/log/slow_query.log Reading mysql slow query log from /var/log/slow_query.log Count: 46 Time=80.46s (3701s) Lock=0.00s (0s) Rows=512311 (117447821), bob[bob]@[10.0.0.32] SELECT * FROM forum_posts Count: 26 Time=68.26s (1775s) Lock=0.00s (0s) Rows=426 (117447821), bob[bob]@[10.0.0.32] SELECT * FROM forum_posts WHERE thread=N Count: 120 Time=3.52s (422s) Lock=0.63s (76s) Rows=58.0 (6960), bob[bob]@[10.0.0.32] SELECT authors FROM forum_posts WHERE lastpost > N ... </pre>
The next step is analyzing this, likely throwing each of these into an EXPLAIN query (or asking yourself why you are selecting every row in the forum_posts table), adding some indexes, and rewriting some code. The scope of this article is finding the bottlenecks… fixing them is left as an exercise for the reader.
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined LogFormat "%h %l %u %t \"%r\" %>s %b" common CustomLog /usr/local/apache/logs/access_log common </pre>
Whichever log format is being used, this is the name at the end of theCustomLogdirective, you’re going to make a copy of thatLogFormat, give it a name like “commondebug”, and switch the CustomLog directive to use this format.
The fields you will be adding are:
*%T– The time taken to server the request in seconds
*%{mod_php_memory_usage}n– Memory used by PHP in bytesLogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %T %{mod_php_memory_usage}n" combineddebug LogFormat "%h %l %u %t \"%r\" %>s %b %T %{mod_php_memory_usage}n" commondebug CustomLog /usr/local/apache/logs/access_log commondebug </pre>
At this point, you’ll be collecting some great data in you apache logs. You can get some good information with some quick shell magic, like so:> awk '{printf("%07d\t%d\t%s\n", $(NF), $(NF-1), $7)}' access_log | sed 's/\?.*//' | sort -g -k1 0001232 0 /baz.php 0001232 0 /bar.php ... 1712160 0 /foo.php 1717640 0 /foo.php 1907800 0 /foo.php 2010840 0 /foo.php </pre>
Replace,-k1with-k2to sort by the delivery times. Keep in mind, delivery times will include the time to send the bytes over the network — http clients can do screwy things, and you’ll occasionally see anomalous data including 120+ second connects where the client just stopped accepting packets, but didn’t close the socket.
From here, you’ll want to examine each of the memory-hogging scripts, and anything which is consitently long-running.
php.ini on their production servers. (This is a good idea, preventing the revelation of any inappopriate information to end users.)php.ini to include lines such as these:error_reporting = E_ALL & ~E_NOTICE ; Show all errors except for notices
display_errors = Off ; Do not print out errors (as a part of the HTML script)
log_errors = On ; Log errors into a log file
error_log = "/var/log/php_error.log" ; log errors to specified file
#!/usr/bin/perl
my $URL_TO_TEST = 'http://test.test/test.test';
my $THRESHOLD = 2;
open(my $log, '>>status_log.txt');
while (1) {
my $start = time();
`curl "$URL_TO_TEST" > /dev/null`;
my $took = time() - $start;
if ($took > $THRESHOLD) {
print $log "$took seconds load ::";
print $log `date`;
print $log "\n\n";
print $log `vmstat -a`;
print $log "\n\n";
print $log `uptime`;
print $log "\n\n";
print $log `ps awux`;
print $log "\n\n";
print $log `mysqladmin -hHOST -uUSER -pPASSWORD processlist`;
print $log "----------------------------------------------\n\n";
}
sleep(30);
}
There is certainly more data you can collect, but the steps above should be enough to start troubleshooting your performance woes. At some point, I should really write about what to do once you’ve found the trouble.
