Dropsafe

by Alec Muffett

Websites that are blocked by O2 at #BarCampBerkshire

Method:

  1. Take the Alexa “Top 1 million” and extract the top 1000 websites
  2. Fetch their homepages as ‘www.site.domain‘ over HTTP using curl

From this list, attached below are the blocked websites; mostly porn, but with some surprises. Notably all blogspot.com blogs are blocked – including the Google security blogs – but not wordpress.com, blogger.com, livejournal.com …

  1. 2ch.net
  2. 4tube.com
  3. 51.la
  4. 888.com
  5. addthis.com
  6. adult-empire.com
  7. adultadworld.com
  8. adultfriendfinder.com
  9. alphaporno.com
  10. amazon.co.jp
  11. anonym.to
  12. baidu.com
  13. beeg.com
  14. bet365.com
  15. betfair.com
  16. bild.de
  17. blogfa.com
  18. blogspot.com
  19. cam4.com
  20. cnzz.com
  21. dmm.co.jp
  22. drtuber.com
  23. empflix.com
  24. ero-advertising.com
  25. extremetube.com
  26. gamefaqs.com
  27. h2porn.com
  28. hardsextube.com
  29. hidemyass.com
  30. imagebam.com
  31. imesh.com
  32. in.com
  33. isohunt.com
  34. istockphoto.com
  35. keezmovies.com
  36. largeporntube.com
  37. linkbucks.com
  38. livejasmin.com
  39. match.com
  40. metacafe.com
  41. mihanblog.com
  42. motherless.com
  43. movie2k.to
  44. myfreecams.com
  45. nate.com
  46. okcupid.com
  47. pch.com
  48. pclady.com.cn
  49. perfectgirls.net
  50. pogo.com
  51. porn.com
  52. pornerbros.com
  53. pornhub.com
  54. pornhublive.com
  55. porntube.com
  56. rakuten.co.jp
  57. rapidshare.com
  58. redtube.com
  59. seznam.cz
  60. slutload.com
  61. spankwire.com
  62. speedbit.com
  63. squidoo.com
  64. streamate.com
  65. stumbleupon.com
  66. sunporno.com
  67. tinypic.com
  68. tmz.com
  69. tnaflix.com
  70. trafficholder.com
  71. tube8.com
  72. tubegalore.com
  73. urbandictionary.com
  74. video-one.com
  75. williamhill.com
  76. xhamster.com
  77. xnxx.com
  78. xtube.com
  79. xvideos.com
  80. y8.com
  81. ya.ru
  82. youjizz.com
  83. youporn.com
  84. yourlust.com
  85. zedo.com

update: removed the links, dan benton is sorta right, and moreover some people might get upset by dns preloading…

⊞

Comments

14 responses to “Websites that are blocked by O2 at #BarCampBerkshire”

  1. Stephen Usher

    Interesting that blogspot and baidu are on there.

  2. Darren Moffat

    And at least three betting sites.

  3. alecm

    The Sun is accessible; Bild is not?

  4. alecm

    “Play Free Games at Y8.com” – ahhhhh

  5. alecm

    …Aaaaaaaand you can use Google Translate to drill through the filter.

  6. alecm

    does block reddit.com/r/nsfw but not reddit.com itself…

  7. Dave Walker

    Mmm – as you say, there are some surprises. I wonder what’s considered “iffy” about Amazon’s Japanese site?

    I’m not on O2 so I can’t investigate by trying to access one of these sites from an O2 link, but is it definitely O2 that’s doing the blocking, rather than some other organisation in the chain? (I’ve just looked, and amazon.co.jp works fine from Vodafone…) Does the blocking look like blackhole routing, or is it just at a DNS level?

    1. alecm

      Apparently Amazon.JP has a reputation for putting porn on the page somewhat unabashedly.

  8. alecm

    An interesting thing about O2 Web blocking (at their corporate HQ, Dave) is that HideMyAss and Anonym.to are blocked, but Tor is not.

    Am wondering whether Tor’s reputation as a “good guy” is protecting them somewhat, as it is immensely embarrassing for a carrier to be discovered to be blocking Tor, as opposed to a commercial service.

  9. Dogsbody

    Do you really want hyperlinks to all hose sites on your blog? I really don’t care but I’d hate to see even less people find your posts due to some crappy web filtering!

    1. alecm

      I’ll take the hit; if there’s an issue I would rather attack the web filtering companies for doing it wrong.

  10. Jake Pearse

    Hi Alec,
    I’ve been testing mobile network blocking for ORG over the past few weeks as part of the work on blocked.org.uk.
    I ran your list through a test script on O2’s mobile network and almost all of the non-porn sites came through ok.
    What I did see was a lot of http 301 and 302 redirects. Did you use the -L flag with curl to follow redirects?

    1. alecm

      Hi Jake,

      No, I did a straight pull of every URL and saved the result; of the 80-odd blocks I sampled perhaps 20 in Chrome and saw the same resulting block page, so I believe my results are realistic.

      This was O2’s Wifi at O2’s headquarters, to I doubt it will be the same as the mobile blocking.

      Is there an issue with my methodology? Shall we document one to act as a future metric?

  11. Jake

    Hi Alec,
    I think my reply got lost so I’ll try again.
    Yes, there must be different filters in place for o2’s wifi and mobile services.
    My method is very similar to yours;
    curl -L or Python urllib2.urlopen.
    Then grep the result for the known values of o2’s blocked pages;
    '^(https://bango.net/).*'
    '^http://wap.o2.co.uk/(18plusaccess).*'

    Thanks,
    Jake

More posts