I’m not an expert programmer, I’m self-taught so I missing bits of info and I haven’t gotten a job doing this. I do this on the side in hopes of making a business out of it or getting a job doing it.
Lootsrc was a website I created and had high hopes for. Lootsrc was a web app that had aggregated PC game prices from various stores. I wrote the crawlers in Nodejs and the website is written in Meteor.
What I Learned
Think about the type of data you will need to be storing.
I’m not sure if it was me or not but I had poor performance from Meteor. From poking around I have seen others have poor issues with it, so I doubt it was all me. This was also back at Meteor 1.3 so I’m sure they made some improvements. Part of this was probably related to not using a relational database when my data needed it. There are many tricks to improving speed the biggest is indexing your database.
Cleaning Data is hard
I didn’t realize how hard it was to clean incoming data and match it with the other data. So many sites have different names for the same game. Every time I think I had it, I would walk away and it would break.
A trick I used with my crawlers was to send me an push notification using Pushbullet API. Be careful if you do this so you don’t flood it. Sending me an push notification when something broke was nice. Email works too but this was my preferred way for when things broke.
Docker and Dokku are awesome
Seriously, if you haven’t heard of Docker or Dokku go check them out. Dokku is like Heroku but open source basically lets you push applications at the command line to your server. It uses docker technology to deploy them on the server. This blog is actually being deployed on a Dokku server.
I’m not trying to knock Meteor or MongoDB there are sites that use and do well. It’s a fantastic tool to quickly get an MVP to test an idea. It was just a poor choice for me. I also don’t feel like Meteor was a great long term tool. Either way, just make sure what tools you’re using fits your needs.
I ultimately shut down Lootsrc because I failed to earn anything from it, the amount of effort and cost it required to keep it up. But it was a great learning experience.
I still think of reviving the project but building it using Elixir and Phoneix. Mainly, just thinking of something to build with Elixir and Phoneix. I doubt I will revive LootSrc though at least in the form it was.
If you haven’t heard Elixir or Phoenix you should go check them out. Phoenix Framework has been doing some incredible things.