Thursday, November 09, 2006

How do I find the geographical location of a host, given its IP address ?

In general, it is impossible - IP addresses are allocated arbitrarily, as there's no inherent connection between an IP address and it's physical location, and there's no reliable method to do the trick.

Yet, doing some detective work could help. Try following methods :




  1. Note the following links for reference :



    A complete list of country codes

    http://www.iana.org/domain-names.htm

    http://www.ics.uci.edu/pub/websoft/wwwstat/country-codes.txt




    A complete list of U.S. state abbreviation

    http://www.usps.gov/ncsc/lookups/abbr_state.txt



    A complete list of airport codes

    http://www.aviationjobsonline.com/airports/citycode.html



    Microsoft's TerraServer - satellites pictures of geographical areas

    http://www.terraserver.microsoft.com/



  2. Use reverse DNS to find out the host's name. This item could supply some clues that could help.




    E.g. given the IP address 132.74.18.2, the command 'nslookup 132.74.18.2' translates the address to construct.haifa.ac.il gives two hints -


    1. The TLD is .il, which hints the host is in Israel.
    2. The next two domains are haifa.ac, so this host belongs to the 'haifa' academical institute (a university, in this case). The Haifa university happens to be in the city Haifa.



    Reverse DNS translation doesnt always work - it depends on the host's [the host with the given IP address] DNS server's correct configuration.



    Another trick is to execute a whois request on the IP address. Try to direct the whois query to whois.arin.net - if it doesn't have the reply it will tell you to query either whois.apnic.net or whois.ripe.net



    Notice that a host in one domain might be hosted in another country. This is due to both virtual hosting, where a domain of a company from one country or region, might be hosted where hosting is cheap.



    Also notice that the .org, .com, and even .edu domains does not imply the host is in the U.S., as many of those domains belong to companies that are either not U.S. based, or are international, and might have some hosts all over the world.



  3. Some hosts support a DNS extension which allows for hosts to enter their geographical location into their DNS record, based on an extension to DNS described in RFC 1876.




    For further information see - http://www.ckdhr.com/dns-loc/



    Another attempt to express a host's geographical location via DNS is done in RFC 1712. Both RFCs define a DNS Resource Record to contain the geographical location.



  4. Visit the host's web server. A web site will sometimes contain hints regarding the site's location.



    E.g. for construct.haifa.ac.il, you can find info at both http://www.haifa.ac.il/ and http://www.ac.il/




  5. Use whois. The whois database contains administrative contact info for all domains, filled in during domain registration time, and updated from time to time. This admin info could give some hints.



    The whois database is not highly reliable - if an address belongs to a large & responsible company, the company will supply reliable info and update it, but as domain name registrators do not insist on keeping the database accurate and current, the data might be incorrect.



    The IP to Lat/Long page will attempt to display the same information in a graphical representation.

    http://cello.cs.uiuc.edu/cgi-bin/slamm/ip2ll/



    The Allwhois.com page allows whois requests for many countries.

    http://www.allwhois.com/




    A list of whois servers, collected by Matt Power, is available at ftp://sipb.mit.edu/pub/whois/whois-servers.list



    Note that the data is usually given for the owners' main branch or contact points, but the IP addresses might be allocated to hosts that may be located at a different location(s).



  6. Use traceroute. The names of the routers through which packets flow from your [or any] host to the host with the given IP address might hint at the geographical path which the packets follow, and at the final destination's physical location.




    E.g. > traceroute www.mit.edu
    traceroute to DANDELION-PATCH.MIT.EDU(18.181.0.31), ...
    1 teg.technion.ac.il (132.68.7.254) 2 ms 1 ms 1 ms
    2 tau-smds.man.ac.il (128.139.210.16) 5 ms 5 ms 5 ms
    3 128.139.198.129 (128.139.198.129) 9 ms 11 ms 13 ms
    4 TAU-shiber.route.ibm.net.il (192.115.73.5) 535 ms 549 ms 513 ms
    5 fe7507.tlv.ibm.net.il (192.116.177.1) 562 ms 596 ms 600 ms
    6 165.87.220.18 (165.87.220.18) 1195 ms1204 ms
    7 nyc28-16-sar1.ny.us.ibm.net (165.87.28.19) 1208 ms1216 ms1233 ms
    8 198.133.27.5 (198.133.27.5) 1210 ms1239 ms1211 ms
    9 sprint-nap.bbnplanet.net (192.157.69.51) 1069 ms1087 ms1122 ms
    10 nyc1-br2.bbnplanet.net (4.0.1.25) 1064 ms1109 ms1061 ms
    11 cambridge1-br1.bbnplanet.net (4.0.1.122) 1185 ms1146 ms1203 ms
    12 cambridge2-br2.bbnplanet.net (4.0.2.26) 1185 ms1159 ms1073 ms
    13 ihtfp.mit.edu (192.233.33.3) 1052 ms 642 ms 658 ms
    14 W20-RTR-FDDI.MIT.EDU (18.168.0.8) 640 ms 665 ms 674 ms
    15 DANDELION-PATCH.MIT.EDU (18.181.0.31) 702 ms 915 ms 868 ms



    The 3rd hop takes the path to the academic network [checked by local whois lookup], the fifth hop takes the path to New-York [on the east coast], and the 10th hop takes the path to Cambridge [in Massachusetts, on the coast, northern to New-York].



    There is a utility named VisualRoute (http://www.visualware.com/visualroute/index.html) which traceroutes a host, and displays the route on a map of the world. The host's location on the map is based on the whois query, which may be wrong - an Israely domain might be displayed as being in Israel though it is hosted in another country.




  7. Some of the services available on the host might give further info.



    E.g. telnet construct.haifa.ac.il 13 <== Time of day service
    Trying 132.74.18.2...
    Connected to construct.haifa.ac.il.
    Escape character is '^]'.
    Wed Jan 21 08:32:53 1998 <== Time difference hints at the
    host's time zone.



  8. Naming conventions of ISPs and back-bones



    AT&T dialups : <port>.<router-location>.<state>.dial-access.att.net




    Port is 2-254 for the dial-up ports, and 1 for the router itself. location: example: "los-angeles-2" (city and router #). state: 2-letter abbreviation.




    uu.net dialups :

    A. <port>.<device>.<city>.<state>.<iu>.uu.net


    B. <port>.<device>.<airport>.<iu>.uu.net




    iu = intended use (meaningless), state is per USPS ZIP code, deviceis Ascend 'TNT' # or Ascend 'MAX' #.


    Ashko

Build An Online Community With Drupal

A Little History

Today, there's a proliferation of tools designed with Web-based communities in mind. Some might argue that the modern community model began with the launch of Slashdot. The code to build a Slash-like community is readily available. Some might point to the Kuro5hin community as the prototype. That code, too, is available for anyone interested in community-building. With varying degrees of administrative and user ease, these tools serve as a useful means to achieve the end of community-building.

Some, myself among them, believe that the real revolution in communities is underway at this very moment. The Web merely delivers the space in which communities form. Increasingly, those communities are themselves becoming movements with greater weight and influence than ever. The American presidential campaign of 2003-2004 may years from now prove to be a watershed moment in the evolution of online communities. MeetUp and Daily Kos are motivating Web users far beyond the Web sphere. They're providing the tools and space within which users motivate themselves and each other toward political activism outside the Web place.

Need further evidence of the value of community-building in politics? Look no further than two American presidential campaigns that clearly "got it," using online community-building to form and foster a stronger political voice than ever before. Though the campaigns didn't last, the impact of the communities they built will be felt for years and elections to come.

One strong example was the Clark Community Network (CCN), a creation of Cameron Barrett and the Wesley Clark for President campaign. Based on modifications to the Kuro5hin tool, Scoop, CCN provided a true two-way communication with Clark supporters and interested observers. Users were allocated their own blog space, with entries voted down or voted to the front page by other members of the community. CCN moved through creation to rollout and more than 5000 active users in less than six weeks. At its peak, the community helped drive the overall Web traffic of the Clark '04 site to well over 175,000 unique visitors a day. That's a powerful forum for both candidate and supporters.

But the best known political community of the 2003-2004 campaign was the DeanSpace site. Based on Drupal, both DeanSpace and its sister, Dean for America, were widely recognized as the communities that put politics squarely at the center of the online map. The focus of the Dean communities was more than mere discussion -- it was mobilization in support of Howard Dean. These communities literally took the American media by storm with their reach and power. That attention, at long last, finally pushed the concept of blogs and communities to the forefront of both technical and popular media reporting.

So, what is this Drupal tool that captured the fancy of even the narrowly-focused American political media? Simply put, it's an easily-customized open source package of PHP code that you, too, can use to create an online community. Regardless of your community needs -- dog-walking, Chinese checkers, Chicago blues, or politics -- Drupal can set your community foundation in place quickly and easily. Let's set the community-values proselytizing aside for a bit, just long enough to paint the basic concepts and technologies with a hands-on brush.
Installing and Configuring Drupal

Drupal bills itself as "community plumbing." You can download the elbow joints and drain traps that are the source code of Drupal at http://drupal.org. Version 4.1.1 is the latest version, released on May 1. The requirements for installation are minimal: the Apache Web server, PHP and a MySQL database installation. These are the recommended tools, though any database server supporting the PHP PEAR libraries will do.

Prior to unpacking the source code, you'll need to check a few PHP configuration settings. In the /etc/php.ini file, assure that you have the following:

magic_quotes_gpc 0
session.save_handler user

PHP4 also provides support for RSS syndication and the Blogger API by default, via the XML extension. If you want to utilize clean URLs in your community, you'll also need to enable the mod_rewrite and .htaccess capabilities on your Apache server.

With the PHP and server configurations set, extract the source code and copy it to the DocumentRoot of the Web server:

tar -zxvf ~/source/drupal-4.1.1.tar.gz
cp -r drupal-4.1.1/* /var/www/html

Next, you'll set up the Drupal database using the mysqladmin tool. The example below uses root to create the database. Adjust your mysqladmin command accordingly. Following the example, the new database will be named "drupal".

mysqladmin -u root -p password create drupal

Now, log in to the MySQL monitor:

mysql -u root -p password

Create the Drupal user permissions on the database. In the example below, "drupal_user" is the database user for whom you're creating permissions. This user need not exist; setting the permissions will create the proper database entry. "localhost" is the local database server and "drupal_pass" is the password you're assigning to the drupal_user account.

GRANT ALL PRIVILEGES on drupal.* to 'drupal_user'@'localhost' identified by 'drupal_pass'

To finish the database setup, flush the privileges and log out of the MySQL monitor with the following commands:

flush privileges;
\q

You can now create the Drupal database tables using the create script provided in the Drupal package. You'll need to change directories into the server DocumentRoot, and execute MySQL, feeding it the database script as input:

cd /var/www/html
mysql -u drupal_user -p drupal_pass < database/database.mysql

With the Drupal database set up, it's time to dig into the configuration file for some minimal site-specific adjustments.

The Drupal installation directory contains an include subdirectory. This is where the configuration file resides. You'll need to set the database directory and the base URL of your site in this file. In your favorite text editor, set the $db_url line of includes/conf.php to:

$db_url = "mysql://drupal_user:drupal_pass@localhost";

This line sets the database directory for your installation. Next, set the base URL by editing the following line:

$base_url = "http://your.url.here";

This is the public address of your Drupal installation. With a mere thirteen small steps, your Drupal installation is browser connection-ready.

The first time you open Drupal, you'll create an account. This first account will become the administrator account for your system. As always, select the username and password carefully.

The administrative features of Drupal present a nearly dizzying array of options. In order to understand those options, it's important to first understand the underlying structural philosophy.
Understanding the Base Drupal Installation

Drupal presents every slice of content attached to the system as a node. This is, as you might have guessed, analogous to a network, in which every desktop, server and printer is a network node. In the case of Drupal, these nodes consist of anything related to the content of the site. The base nodes of the Drupal system include many pieces such as title, author, body, comments, votes, score, users, and authored on date. Other node types include polls, static pages, personal blog entries, forum topics, and book pages. Collectively, these discrete pieces form the core of the Drupal system.

In parallel to Drupal's node system is its scheme of blocks. Think of these blocks as the visible user and administrative tools. Blocks are a gateway for you (as admin) and your users to access additional tools or view information about the system. These blocks include login, navigation, most recent poll, who's online, who's new, and blogs. If you bring to your Drupal administration some PHP experience, it's an easy task to create blocks specific to the purpose of your community. In fact, the Drupal authors have provided a wealth of documentation to assist you to do so. We'll look at a few examples in a bit.

Nodes and blocks alone make Drupal a powerful, easily configurable community-building tool. But, there's yet another level to the system. The heavy-lifting behind the scenes of the Drupal installation is the module. Modules, in fact, control both blocks nodes. They extend the base functionality of the Drupal installation. Modules can be enabled or disabled and protected with user-appropriate permissions. Critical modules include:

* admin - Provides all the administrative features
* block - Controls the boxes around the main content
* blog - Creates personal blog space for registered site users
* user - Provides the ability for users to register and log in
* help - Controls a deep and very useful help system
* node - The core module that allows content submission to the site
* system - Allows full administration of the site
* profile - Provides configurable user profiles
* tracker - Tracks recent posts to the system

The combination of nodes, blocks and modules is a powerful one. It provides user and administrative granularity unsurpassed in other community tools. With PHP behind the scenes and a MySQL backend, the configuration options are nearly endless. On your first login to the newly installed Drupal system, it's well worth your while to look carefully through the administrative and configuration options. You'll be startled by how much control lies at your fingertips.
Customizing Drupal For Your Community

Exactly what type of community do you want to build? Are politics the crackers in your soup? Do you lean more toward creative activities; writing, art, music? Each community type will have a different set of feature requirements. Political communities, for example, should always provide forums, hierarchical commenting and polls. You may also want to send new post information to the blog aggregators such as weblogs.com. These are features that can be enabled as modules in Drupal.

While political communities require high levels of user interaction, a community focused on more creative endeavours might need only light commenting and the ability to collaborate on community works. In a community of this sort, "extra" features can be disabled with a single mouse-click in Drupal. It's important to clearly think through your community needs prior to building out a community site with too few or far too many features.

For all the pre-planning, the final voice on features in your community will be the users. The flexibility of Drupal and its full default set of features make it easy to fulfill the needs of those users with minimal effort.

Occasionally, though, you or your users will find a need for a feature that can't be met by the default Drupal tools. If you're proficient in PHP, you can easily design, test and add modules to your installation. The Drupal authors have provided a wealth of instruction and guidance for creating these modules. Of interest to your coding are the following:

* Writing Themeable Modules
* Writing a Node Module
* Using Profile Data and Writing Custom PHP Pages

If you're more interested in providing the features than coding them, you may find that someone else has already done the tough work. Sites providing downloadable Drupal modules include:

* Sourceforge.net
* DeanSpace
* BiteSizeInc
* Cortex Communications

Paying close attention to your user requirements and making the modifications to meet those needs will help assure the growth of your community.
Conclusion

The Web as a place? It's been that since the beginning. Email, chat, and instant messaging all contribute to the sense of place on the Web. They serve as gathering places where a broad collection of voices, given proper time and care, become one. The surge of Web communities, in wikis and blogs and political sites, is really just a return to roots. It's one that's taken even the mainstream media by storm. That surge has provided yet another glimpse of the human potential of the Web.

And it's a surge spurred, in part, by tools; tools like Drupal. With a little care and minimal configuration, you too can install the plumbing for your Web-based community.

Lessons in Javascript Performance Optimisation: 90 seconds down to 3 seconds

I've recently been optimising the guts out of a JS webapp I wrote, which was making IE crawl to a halt. I discovered this after introducing a stress-inducing data set. (Using Rails' fixtures makes light work of this; since the fixtures are Ruby templates just like the web templates, it's easy to introduce a loop to create lots of data.)

With a rather large data set (100+ items, each several fields), IE would take about 90 seconds to churn through the initial script before the user could do anything. Firefox would run the same thing in about 8 seconds, still too long for a web page, but incredibly about ten times as fast as IE. I'm wanting to avoid pagination at this stage, so first priority was to tweak performance and see if we can keep everything on the same page.

After some sophisticated profiling ((new Date()).getTime():D), the main culprit was revealed to be prototype's $$. It's a fantastic function, but if you try to grab all elements belonging to a certain class, and the DOM is really big, $$(".cssClassName") can be slow. *REALLY SLOW* in IE. Remedy:

  • Removed trivial usages of $$() - e.g. in one case, the script was using it as a simple shorthand for a couple of DOM elements, and it was easy enough to hardcode the array. i.e. $$(".instruction") becomes [$("initialInstruction"), $("finalInstruction")]. The former notation is cuter, but unfortunately impractical on a large web page.
  • Introduced the unofficial selector addon. Seems to have improved performance in more complex queries, i.e. $("#knownId .message"), but doesn't seem to have affected performance of $$(".classname").
  • Finally, I bit the bullet and scrapped $$(".classname") altogether. It's more work, but the script now maintains the list of elements manually. Whenever an element is added or removed, the array must be adjusted. Furthermore, even the initialisation avoids using $$(), thanks to some server-side generated JS that explicitly declares the initial list of elements belonging to the class (i.e. the list that would normally be returned by $$()). To do this, the following function is called from onload(), generated with RHTML.
PLAIN TEXT
JAVASCRIPT:
  1. function findAllItems() {
  2. <% js_array = @items.map { |item| "document.getElementById('item#{item.id}'),"}.join
  3. js_array = js_array[0..-2] if @items.length>0 # Drop extra comma at end -%>
  4. return [<%= js_array %>];
  5. }

The last step explicitly identifies all items in the class, removing the need to discover them by traversing the DOM. I wasn't really sure how much time it would save - after all, you still have to look the elements up in the DOM and assign them to the array. But when I tried it, the savings were supreme - on IE, from around 45 seconds to about 2 seconds.

I have also incorporated Dean Edwards' superb onload replacement to get the ball rolling before images are loaded. It's a neat trick and takes 5 minutes to refactor it in.