As teh author of the worlds least popular news aggregator I present to you a new and improved technology.

But first this....

Prefetch

Note: I don't want this to be here but it needs to be here.

I really like RSS and Atom feeds because they allow me to keep track of many thousands of blogs and other news outlets without involving 3rd parties who some how end up being responsible for what I read.

I think people fail to aperciate the absurdity of involving undesirable 3rd parties in 2 party interactions.

It should be that one person is writing shit while a second may chose to read it. This model should be sacred. You should be filled with rage and start throwing large objects around when it is violated.

When you talk with friends, relatives or anyone else there might be people listening in depending on the location you chose. People might interrupt your conversation to add their thoughts without prior warning.

Online, when involving a 3rd party you don't get to talk with anyone directly. You talk to a multinational as-if they are the person, the multinational records and moderates your speech using any agenda they like, then they relay your message to the person you had intended it for OR NOT.

One doesn't simply share stuff with others eventho the interface might suggest it. It might very well be that you have to pay to reach your subscribers. People are not subscribed to you, they are subscribed to the recorded, moderated version of you as presented (sold) by the multinational.

Big data allows the multinational (and those it sells your personal information to) to reveal things about you that you might not be aware of yourself.

Your friends might have an above average interest in festivals, your ex might have talked about wards one time, your aunt might drink a lot WHILE 100% of other users who share that specific profile have a poor attention span while earning a below average income. You have 72 other character patterns putting you in that same category.

A long story short, that job application in 2036 where you didn't get invited for an interview - this was why. You might ignorantly wonder why your grand kids didn't get into that university but it was all in that data you shared. You wanted to invite your friends to some political rally but the message never went though?

It all sounds highly conspiratorial but look at it from the other angle. If someone can buy huge influence over millions of people like you for a few thousand bucks. Why would they turn down such an offer?

What chance does a candidate have in an election if no one talks about him? Could it be that that is worth some money? Have you noticed how politicians are increasingly screwing over the public? (their voter) Do you think the proverbial "big money" having the power to give you opinions and take them away from you might have something to do with that?

Do you know what little influence you have over the system in this day and age? Trust me, you cant afford to lose bits of it. You are much to close to zero influence to lose any more.

You also know that marketing messages are designed to get to you. If you give them all of your personal info as well as the means to engineer your world view these messages will cut straight to your core. The same marketing effort might appeal elsewhere to those holding the exact opposite world view. They will totally present their product in ways that would make you rage - but you will never see that version.

I don't actually have to prove any of this nor do you need to believe it.

We can just agree that that article you are writing should be available to the people who want to read that article. Not your mum, not your employer, not your government, not the tech multinational, not the military contractor, not one other person should have a say in it. Would we allow the 3rd party influence your personality would change and you would learn to apply self censorship. The author that is you would essentially be dead.

RSS and Atom

Your blog might already have an RSS feed and people might already be using it to keep track of your new publications. Likewise, those few websites you frequently visit might all have feeds that you can aggregate yourself.

An aggregator is a program that collects related items of content and displays them or links to them. An RSS aggregator is an aggregator that loads RSS and Atom feeds and displays their content in a convenient way. The user, in stead of visiting [many] websites to find something interesting to read, is presented with a simple interface that displays content from dozens, hundreds, thousands or even millions of websites.

This, for example is RSSOwl on Mac.

RSSOwl is a desktop application for Linux, Windos and Mac. On the left we see folders containing feeds. On the right we see at the top the headlines and at the bottom the article content. One clicks the feed or the folder and the headlines therein are displayed for you to click on.

This is (my) Salamisushi RSS Aggregator

It is a Greasemonkey / Tampermonkey script that runs in the browser. There are no folders and there is no article reading area, in stead it has links to new news headlines.

The major difference with RSSOwl is that it aggressively discards all data that isn't mission critical. There is no giant database with news articles, in stead it has a sorted array that only contains 1) publication date, 2) article title, 3) link to article, 4) article source and only for articles that are new enough. If you configure it for 1000 results it will discard anything older than the oldest item.

RSSOwl keeps lots of information about each feed and has lots of nice options adding to the data set. As a desktop application it probably loads the feeds considerably faster. Its great if you have a few hundred subscriptions or if you want to build a list of headlines for your feeds. It will work with a thousand feeds but it will need to store all that data in its database.

My aggregator will work with any number of feeds. You simply provide flat lists of feeds or OPML files and it will load them one by one, load all feeds in them and move on to the next feed list. It has roughly 20-40 thousand feeds preconfigured. Easily the largest curated list of RSS feeds on the web.

It can also publish the results to a website (using http post with a password) which looks like this in my implementation:

http://news.go-here.nl/

Now there are 2 issues I run into in my hunger for moar.

1. RSS feeds can be rather large if you are trying to download and parse thousands of them.

2. The published list of new news is not really machine consumable.

For this I coined: NNO and ESS

You may read the spec here if you so desire (and I hope you do)

Even Simpler Syndication [ESS] and No Nonsense Outline [NNO]

The purpose of ESS is to offer aggregators a dumb api so that they can gain insight into the publication date of rss and atom feed items. Like RSS is a syndication of a websites news articles that has a good number of features ESS is a syndication of RSS that lacks all but the bare necessities. ESS files are typically (much) smaller than 1 kb. They are text documents and parsing is incredibly fast because they start with the time stamp at which the newest article was published. If this is the same time stamp as the previous session or if the newest article is to old to make the cut the aggregator can drop it and move on to the next.

ESS is structured like this:

buildTime, feedUrl,
minutesBeforeBuild, titleDumbChecksum,
minutesBeforeBuild, titleDumbChecksum,
minutesBeforeBuild, titleDumbChecksum,
minutesBeforeBuild, titleDumbChecksum

Where:

buildTime = The build time of the ESS in seconds (rounded down to be divisable by 60)
feedUrl = A link to the RSS or Atom feed described by the ESS
minutesBeforeBuild = The publication date in minutes before buildTime
titleDumbChecksum = A checksum in case the feed has multiple publications sharing the same date

(read the full spec)

The purpose of NNO files is primarily in sharing. After you've parsed god knows how many feeds you end up with a relatively short list of fresh news articles. You reduce this list to simple feed urls and a somewhat exotic time stamp then publish it on the web. These files are also incredibly small. Much smaller than the time and link list they represent.

Your friends or other audience can obtain your NNO file as well as any number of other NNO files, sort them by date, eliminate older articles, and their client can load and display the tiny number of newest headlines from the RSS feeds. When they scroll down the page additional feeds can be loaded dynamically and their headlines displayed.

If they also have their own subscriptions their agregator can proceed to load its own feeds and mix them with yours into a new time sorted list.

THEN, (and here it gets really exiting!), they can publish their own NNO file that is both newer and enriched with their own subscriptions.

Rinse lather repeat!

And there you have it! A whole new internet!

You are probably visiting websites looking for new things while you could have been using RSS and use a fraction of time to keep track of a hundred of them.

With my new microformats you could extend your subscription list to include every website you've ever visited and you could subscribe to interesting peoples subscription lists to extend your news gathering to the millions of websites.

I know this doesn't sound very interesting but trust me, the people behind those websites you visit put a lot of effort into writing articles. You ended up reading one of them (most likely though a search engine) the ods these webmasters publish other articles that you find interesting is truly enormous. If they were not able to produce interesting content consistently their article wouldn't even be in the search engine.

If you end up on 3 unique websites per day on average that could add up to a thousand subscriptions per year.

They might not all publish every day but you can expect that list of subscriptions to produce at least 200 articles that are all trying to appeal to the audience that is almost exactly you if not exactly. Out of these 200 articles, with a bit of headline gazing and a bit of scrolling you are going to find something truly worth reading every day.

I do keep a lengthy blacklist of websites that wont make it though my aggregations as well as a huge bad word and bad word combination list.

This is necessary because most people visit a tiny number of websites manually. Because of that the webmaster attempts to be their source of all news which results in thousands of articles informing the world that David Bowie died. Filtering out that type of articles (for a few days) helps me get at the other news but also gives me a good impression what everyone else is reading.

You are all reading western publications about a very narrow spectrum of topics. There are 195 countries in the world and from a popular news perspective it seems a lot like nothing ever happens in most of them.

There are over a billion websites on the internet. I cant say my 40 000 subscriptions provide a meaningful insight into what was published today. It looks like you are all going to have to help me crank up this number into the millions. I can dream right?

Surely you would all prefer to all be puppets of the establishment and parrot Trumps latest tweet as if it means something?

I know, I know, it is going to take some minor effort to help me make the ESS and NNO revolution happen.

I tried to do most of the work for you. As a PHP man I oddly started by writing a Node.js module.

https://gist.github.com/gaby-de-wilde/341a0d1897e47b57af39fd0a2a06b718

The module returns the ESS and has 2 arguments:

1. The link to the RSS or Atom feed described.
2. An array of objects formatted like this:

[
{ pubDate: 1503674340000, title: "hello world"},
{ pubDate: 1503674220000, title: "foo" },
{ pubDate: 1503674160000, title: "bar" }
]

install with:

npm i -S gist:341a0d1897e47b57af39fd0a2a06b718

Autodiscovery can be had by adding this to the head of your html documents.

<link rel="alternate" type="application/ess" 
  title="example" 
  href="/ess/" />

Feel free to post your ESS feed in the comments and I will be sure to subscribe to it.

I will write the other implementations when I find the time to do so but it shouldn't be that hard to make your own.

Thanks for reading :-)

Ill end by posting a few gems from the harvested results.

The Creative World’s Bullshit Industrial Complex
http://99u.com/articles/55974/the-creative-worlds-bullshit-industrial-complex

China’s planned supersonic maglev transport in evacuated tubes
https://www.nextbigfuture.com/2017/09/chinas-planned-supersonic-maglev-transport-in-evacuated-tubes.html

Finding Better Wind Energy Potential
http://solarthermalmagazine.com/2017/09/06/finding-better-wind-energy-potential/

Why these scientists argue that we are turning Earth into an alien planet
https://www.geekwire.com/2017/scientists-argue-turning-earth-alien-planet/

Nature Offers Solutions to Water Woes and Flood Risks
https://www.ecowatch.com/suzuki-flooding-nature-2482221066.html

Czech PM candidate Babis to face fraud charges after vote
http://www.oann.com/czech-pm-candidate-babis-asks-parliament-to-let-him-rebut-fraud-charges/

Predatory journals a global problem
https://phys.org/news/2017-09-predatory-journals-global-problem.html

Irma takes dead aim at Florida; 1.4M people ordered to flee
http://www.thetruthseeker.co.uk/?p=157464

(for laughs)
Just When You Thought Southeast Asia’s Democratic Regression Couldn’t Get Any Worse …
https://www.cfr.org/blog/just-when-you-thought-southeast-asias-democratic-regression-couldnt-get-any-worse-0

New Liquid-Metal Membrane May Make Hydrogen Fuel Cell Vehicles Viable
http://www.engineering.com/DesignerEdge/DesignerEdgeArticles/ArticleID/15590/New-Liquid-Metal-Membrane-May-Make-Hydrogen-Fuel-Cell-Vehicles-Viable.aspx

Russian mafia encroaching on US mafia!
http://adscam.typepad.com/my_weblog/2017/09/russian-mafia-encroaching-on-us-mafia.html

The Houston Flooding Pushed the Earth's Crust Down 2 Centimeters. The 275 trillion pounds of water from Hurricane Harvey deformed the ground in Texas.
https://www.theatlantic.com/technology/archive/2017/09/hurricane-harvey-deformed-the-earths-crust-around-houston/538866/

Western Australia joins two-thirds of country to ban fracking
http://tapnewswire.com/2017/09/western-australia-joins-two-thirds-of-country-to-ban-fracking/

Mouse Study Shows a Ketogenic Diet Increases Longevity, Strength
https://scienceblog.com/496122/mouse-study-shows-ketogenic-diet-increases-longevity-strength/

Who Has Been ‘Begging for War’?
https://www.antiwar.com/blog/2017/09/05/who-has-been-begging-for-war/

And so on!