tag:blogger.com,1999:blog-7346520062335584992.post6536862248220855389..comments2011-03-14T12:42:13.399+13:00National Library of New Zealandhttp://www.blogger.com/profile/05067703181520460430noreply@blogger.comBlogger23125tag:blogger.com,1999:blog-7346520062335584992.post-71414520664901745442008-10-21T14:15:00.000+13:002008-10-21T14:15:00.000+13:00Hi Andrew:<BR/><BR/>We've posted a response to a couple more frequently asked questions to the <A HREF="http://www.natlib.govt.nz/about-us/news/20-october-2008-web-harvest-faqs" REL="nofollow">FAQ</A>, including explaining why the harvester is off-shore and why we didn't notify webmasters in advance. I think these are both areas for improvement should we do another harvest.<BR/><BR/>GordonGordon Paynterhttp://www.blogger.com/profile/13375515204887559709noreply@blogger.comtag:blogger.com,1999:blog-7346520062335584992.post-29547564897317469802008-10-21T14:12:00.000+13:002008-10-21T14:12:00.000+13:00Sigh... If they crawl more than a couple of GB from my site, or if they take down the server, they will be receiving an invoice from me for the international bandwidth used and for loss of business. Robots.txt is there for a reason!!!Jason Franksnoreply@blogger.comtag:blogger.com,1999:blog-7346520062335584992.post-38981456310288478592008-10-21T14:09:00.000+13:002008-10-21T14:09:00.000+13:00Hi anonymous:<BR/><BR/>You note that <I>"the Internet Archives Wayback Index) to look up the names of hosts that are physically in New Zealand"</I> does not seem possible... and its not.<BR/><BR/>Something got mixed up, and this is meant to refer to <I>the Alexa web search API</I>. We'll update the page.<BR/><BR/>GordonGordon Paynterhttp://www.blogger.com/profile/13375515204887559709noreply@blogger.comtag:blogger.com,1999:blog-7346520062335584992.post-63938422727467798402008-10-21T10:47:00.000+13:002008-10-21T10:47:00.000+13:00@boris - aha, there was discussion ... darn, missed it at the time - did anyone submit?Mike Riversdalehttp://www.blogger.com/profile/00112999693425305730noreply@blogger.comtag:blogger.com,1999:blog-7346520062335584992.post-72263682808405277282008-10-21T10:45:00.000+13:002008-10-21T10:45:00.000+13:00Thanks for being a part of this obviously thorny discussion and for adding to the FAQ.<BR/><BR/>When you guys and gals come to do the next crawl maybe some prior discussion and heads-up would alleviate some of the slightly bad feeling being generated. Working together is always better than finding out after the event.Mike Riversdalehttp://www.blogger.com/profile/00112999693425305730noreply@blogger.comtag:blogger.com,1999:blog-7346520062335584992.post-60696224755216428482008-10-20T21:41:00.000+13:002008-10-20T21:41:00.000+13:00It's very disappointing that National Library is choosing to ignore good internet citizenship in order to achieve its ends. It's disingenuous to ask "how we can make it better for you" when electronic documents are being collected with the latent threat of a $5,000 fine for any publisher that does not comply with making the documents available (see s40 of the National Library Act 2003, http://legislation.govt.nz/act/public/2003/0019/latest/DLM192266.html?search=ts_all%40act%40bill%40regulation_national+library ).<BR/><BR/>I'll co-operate because I can't afford that fine, but don't expect me to like it. I note that the Internet Archive, to their credit, honours robots.txt (http://www.archive.org/about/exclude.php )Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-7346520062335584992.post-65637881267112031522008-10-20T21:14:00.000+13:002008-10-20T21:14:00.000+13:00I guess we must remember that NLNZ has the legal right to collect your site - in full<BR/><BR/>You had a chance to have a conversation with them about that in the consulation that occured about <B>National Library of New Zealand (Te Puna Mātauranga o Aotearoa) Act 2003</B> (see <A HREF="http://www.legislation.govt.nz/act/public/2003/0019/latest/whole.html?search=ts_act_National+Library+of+New+Zealand+(Te+Puna+M%C4%81tauranga+o+Aotearoa)+Act+2003#DLM191962" REL="nofollow">http://www.legislation.govt.nz/act/public/2003/0019/latest/whole.html?search=ts_act_National+Library+of+New+Zealand+(Te+Puna+M%C4%81tauranga+o+Aotearoa)+Act+2003#DLM191962</A><BR/><BR/>I guess the conversations and submissions are buried in a vault - but sometime will be unveiled along with the various harvests that the nlnz is doingBorisnoreply@blogger.comtag:blogger.com,1999:blog-7346520062335584992.post-1909200275604438342008-10-20T18:59:00.000+13:002008-10-20T18:59:00.000+13:00If you're picking on sites physically hosted in NZ then scraping them from a source address that was also within New Zealand would make it a lot easier for some of us. Not to mention cheaper.<BR/><BR/>I know that National Library has a Citylink connection. I can't understand why this is not using it? Why are you exporting all of our content to San Jose, and charging us for that?<BR/><BR/>Cheers,<BR/>Andrew.Andrew McMillanhttp://andrew.mcmillan.net.nz/noreply@blogger.comtag:blogger.com,1999:blog-7346520062335584992.post-45093202902004396582008-10-20T17:44:00.000+13:002008-10-20T17:44:00.000+13:00October 16, 2008 at 9:12 PM, Tim Snadden said... <BR/>"<I>You say 'let us know, we'll make it stop'. By creating a robots.txt file the site owner *has* let you know and you are ignoring their wishes.</I>"<BR/><BR/>As a web publisher myself, I whole-heartedly agree with this viewpoint. The "Robots.txt" file is there to control release of information, prevent direct grabbing of pages and files NOT for general distribution and indexing, and basically to let 'agents' know what they are allowed to grab and what they are required to leave alone.<BR/><BR/>I for one will be blocking your harvesting bot directly, even if it means turning down your 'offer' of posterity.<BR/><BR/>Also, it would be prudent to allow the 'harvestees' to request that you delete any harvested content, since it was been scraped without implicit consent.<BR/><BR/>Thank you for bringing to light your utter disregard of our wishes.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-7346520062335584992.post-52585282441758391892008-10-20T17:05:00.000+13:002008-10-20T17:05:00.000+13:002. The crawl engineers used several available services (eg the Internet Archives Wayback Index) to look up the names of hosts that are physically in New Zealand but not registered in the nz domain.<BR/><BR/>You state this, but in my mind this doesn't actually sound possible.<BR/><BR/>How can the internet archives wayback index tell you if a host is in New Zealand or not?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-7346520062335584992.post-71661038039722260842008-10-20T16:55:00.000+13:002008-10-20T16:55:00.000+13:00Link to the <A HREF="http://www.natlib.govt.nz/about-us/news/20-october-2008-web-harvest-faqs" REL="nofollow">Web Harvest FAQs</A>.Courtney Johnstonhttp://www.blogger.com/profile/13465703476413455843noreply@blogger.comtag:blogger.com,1999:blog-7346520062335584992.post-87092922282431344152008-10-20T16:32:00.001+13:002008-10-20T16:32:00.001+13:00Hi Dave: <BR/><BR/>I have also answered your question about very large sites directly in the FAQ. To provide a little more detail, we are trying to make the crawl broad rather than deep, so if your website is very large, then the chances are we won’t capture it all. <BR/><BR/>However, we don’t yet know how deep we can get into large websites. Consider this: we’re currently aiming to harvest 100 million URLs from just over 300,000 hosts. On the face of it that’s an average of 333 URLs per host. However, a lot of hosts will be empty, or redirects, or small. Here’s another case study though: in a recent domain harvest the BNF (French National Library) about half the .fr domains had 10 URLs or less, and only about 0.04% were crawled beyond 10,000 URLs (see “Legal deposit of the French Web” on http://iwaw.net/08/). It therefore seems unlikely we will be harvesting 10s of millions of URLs from your servers.<BR/><BR/>GordonGordon Paynterhttp://www.blogger.com/profile/13375515204887559709noreply@blogger.comtag:blogger.com,1999:blog-7346520062335584992.post-58422799283438810562008-10-20T16:32:00.000+13:002008-10-20T16:32:00.000+13:00Hi Mike and Mike:<BR/><BR/>I’ve added your questions (about where the list of names comes from, and how we find websites hosted inside and outside New Zealand) to a FAQ page, which you can find here:<BR/>http://www.natlib.govt.nz/about-us/news/20-october-2008-web-harvest-faqs<BR/><BR/>It’s all a bit complex, but I hope that covers it off.<BR/><BR/>GordonGordon Paynterhttp://www.blogger.com/profile/13375515204887559709noreply@blogger.comtag:blogger.com,1999:blog-7346520062335584992.post-77368173492495441092008-10-18T03:11:00.000+13:002008-10-18T03:11:00.000+13:00interesting the non response to issues raised hereI see aship in the harbournoreply@blogger.comtag:blogger.com,1999:blog-7346520062335584992.post-82301743733270322272008-10-17T09:04:00.000+13:002008-10-17T09:04:00.000+13:00I don't think trying to apply a paper model to the internet is practical.<BR/><BR/>I run several large NZ websites containing literally tens of millions of indexable pages and unknown quantities of dynamic pages (search results etc). Thousands of pages change daily and total content is several hundred gigabyte, and lot of that is video and imagery.<BR/><BR/>Do you intend to download all of the content from all of my sites? (they are all withing the .co.nz)davehttp://www.blogger.com/profile/05654861944179220597noreply@blogger.comtag:blogger.com,1999:blog-7346520062335584992.post-10973874390487793282008-10-16T22:06:00.000+13:002008-10-16T22:06:00.000+13:00Gordon's left for the day, but he'll post a reply to your question tomorrow morning.<BR/><BR/>maybe Gordon could get the internet at homeno sleep till midnightnoreply@blogger.comtag:blogger.com,1999:blog-7346520062335584992.post-85117912364206250122008-10-16T21:12:00.000+13:002008-10-16T21:12:00.000+13:00You say 'let us know, we'll make it stop'. By creating a robots.txt file the site owner *has* let you know and you are ignoring their wishes. Bad decision, bad net citizen.Tim Snaddennoreply@blogger.comtag:blogger.com,1999:blog-7346520062335584992.post-9170217299625282872008-10-16T20:01:00.000+13:002008-10-16T20:01:00.000+13:00I think Mike F has asked the question I was trying to ask - for instance my work site (http://www.miramarmike.co.nz) is actually a Google hosted platform in (I assume) the US. And the opposite probably applies - international content hosted in NZ - but on a smaller scale.<BR/><BR/>And, of course, NZ content that's on *any* site in the world - Flickr immediately springs to mind (discussions as well as the actual photos)<BR/><BR/>Back to "putting it onto the Web is legally publication", I'm not sure that answers the question. If the rule is do not index this (using robots.txt) then I don't see how "it's published" changes it. This isn't books, this is the web and the rule (de fact, understood, community) rule still applies.<BR/><BR/>But hey, I also understand you've a civic duty to perform and appreciate the dilemma and that you're open to talking about it.Mike Riversdalehttp://www.blogger.com/profile/00112999693425305730noreply@blogger.comtag:blogger.com,1999:blog-7346520062335584992.post-16907542953254739862008-10-16T17:26:00.000+13:002008-10-16T17:26:00.000+13:00Well, either.<BR/><BR/>I guess i'm wondering <BR/>a) how you're getting the list of names to harvest, <BR/>and <BR/>b) how you tell the difference between sites hosted in new zealand, and those hosted overseas with .nz domain names.mike forbeshttp://www.blogger.com/profile/10429321058652694673noreply@blogger.comtag:blogger.com,1999:blog-7346520062335584992.post-13349008807208897472008-10-16T17:22:00.000+13:002008-10-16T17:22:00.000+13:00hey Mike<BR/><BR/>Gordon's left for the day, but he'll post a reply to your question tomorrow morning. Just to clarify - the list of domains; do you mean all the ones that are being harvested, or the not-hosted-in-NZ ones?Courtney Johnstonhttp://www.blogger.com/profile/13465703476413455843noreply@blogger.comtag:blogger.com,1999:blog-7346520062335584992.post-83821268563145219392008-10-16T17:14:00.000+13:002008-10-16T17:14:00.000+13:00I'm curious as to how you are collecting these domains..<BR/><BR/>where are you getting the list from?mike forbeshttp://www.blogger.com/profile/10429321058652694673noreply@blogger.comtag:blogger.com,1999:blog-7346520062335584992.post-60949785720544633752008-10-16T17:03:00.000+13:002008-10-16T17:03:00.000+13:00Hi Mike:<BR/><BR/>Yes, in many cases it is similar to collecting people's diaries (national libraries do that too). One of the interesting ethical issues that web harvesting throws up is that legally posting stuff on the web is seen as "publication", the people doing it might think of it more as "communication", and expect some level of privacy. We will have to think about this more before we make the harvest publicly available, but for now our focus is capturing as much at-risk material as we can.<BR/><BR/>We are gathering a small sample of NZ content that is not hosted in New Zealand, but it is very difficult to detect this reliably in an automated fashion. Our selection is therefore hand-vetted. The Czech national library is doing some neat work on automating this process using whois lookups, and looking for Czech phone numbers, names, language and email addresses on web pages that we might be able to exploit in the future (if we ever do another harvest).<BR/><BR/>GordonGordon Paynternoreply@blogger.comtag:blogger.com,1999:blog-7346520062335584992.post-54357546425155243622008-10-16T16:36:00.000+13:002008-10-16T16:36:00.000+13:00Interesting stance.<BR/>Isn't that a little like saying, "We want to keep a record of people's physcaly diaries and therefore can read over people's shoulders"?<BR/><BR/>I get and suppotr the intention.<BR/>However by breaking a fundamental rule of the WWW will probably mean you'll get blocked by more direct ways now and in the future.<BR/><BR/>Oh, and how are you handling NZ content that is not hosted in NZ? Just curious.Mike Riversdalehttp://www.blogger.com/profile/00112999693425305730noreply@blogger.com