Pokémon Go Bots (History)

I’ve written the project up here as a kind of history, tracking the increasing sophistication of the apps and the things I learnt along the way. It’s lengthy, so I bolded the important bits for skimming. Project links, with initial release dates:

• The original personal bot (15 Mar): https://pastebin.com/cBBT4wB6
• The final personal bot (13 May): https://github.com/Pyorot/Percy
• The current public bot (6 Dec): https://github.com/Pyorot/Abyo

Percy

In the beginning, I knew nothing about JavaScript, except that you can put it in your webpage and make someone’s browser generate 100 message boxes do interactive things. I started by loading dev tools in a browser tab of the map, finding the console, looking for the data, then implementing some basic filtering logic, copy+pasting foreign JS syntax from the internet. I searched for an app to handle push notifications, found PushBullet, and copied an XmlHttpRequest from somewhere. I looped the code and the result was Percy.

The initial version was fully procedural (to the extent that there were no subroutines at all, only one main routine!). But at the time, I was pleased enough to annotate it and display it as a cool quick tool I made with no JS knowledge that might inspire someone else to try a similar thing. As I learnt more JS and more and more users started to give feedback, I made various improvements, leading to the final version, which is a very simple, elegant and self-explanatory piece of code.

The changes included:

• More modular code. Making it much easier to test.
• More general filter syntax. People often wanted to set multiple criteria for each Pokemon species, which eventually led to me imagining a more powerful syntax.
• Sending queue. To avoid rate limits, I ensured not too many messages were sent at a time by making a function that sent the first item in a queue, then called itself on the rest of the queue with a fixed timeout.
• Harnessing more data and routines from the website. I gave the Pokémon names as opposed to IDs, more robustly checked if they had already been processed, and fetched their time until despawn, all using functions and data that were already availabile in the map.

Pyobot (Public)

Percy is very DIY; it requires copying+editing code to use and running a laptop constantly. I decided therefore to adapt it to post to public Discord channels – the management team decides what will be posted to them, and users subscribe as they wish. This was my version of Pyobot (whose code is very inelegant, owing to inter alia an old inexplicable filter syntax with a complicated flowchart that I lost and error handling that mostly doesn’t work).

This led to many new challenges:

• Location information. Percy sent distances from a fixed (home, or work, or…) location with each alert. This was obviously useless in a public channel, so I tagged each alert instead with location information – postcode and suburb, pulled from free online reverse geocoding APIs like OpenStreetMap Nominatim. I also embedded a Google Static Maps image for each alert!
• Remote updating of settings. After failed research into VPSs, I found somebody with an always-on computer to run the code. I wanted to maintain the flexibility of being able to e.g. quickly design and implement new channels, so I designed the bot to periodically download a JSON file with filter settings etc. from JSONBlob, where I could edit it.
• Known-spawn cleaning. With the volume of alerts sent, it seemed sensible to periodically clean despawned Pokemon from the known list, which each new spawn was checked against to prevent multiple alerts while it was active.
• Error handling. Lots of APIs kept erroring, leading to avoidable situations like a spawn getting dropped before being sent because the location annotation APIs failed. I tried to fix this with try/catch as usual, being very new to the idea of asynchronicity, and found I mostly failed, and didn’t really resolve it until my later redesign of the bot. I settled for synchronous HTTP requests, whose hackish error-handling worked on some browsers but not others…

So, though I didn’t resolve the last of these challenges, Pyobot became very popular and useful to the Pokémon GO community, and led to increased uptake of Percy for customised personal filtering. Not long afterwards…

Pyobot (by Moriakaice)

A professional web developer came along (late April), was amused by the fact that I was running a public bot on the laptop of somebody who was always on the verge of getting banned from the community, told me about Node.js and “using the right tools for the job”, ported the bot to Node and took over running it on his VPS. The same day his port was ready, laptop guy was banned, by complete coincidence.

The improvements he made escalated from there:

• He obsoleted Percy by implementing a database and Discord bot interface that allowed users to directly message the bot account, send custom setting commands, and receive alerts. Crucially, people no longer needed to run it on their own laptops.
• He implemented another database to collect data of historical spawns and generate heatmaps.
• Errors were actually being handled now!

I was happy with how the project was going, so stepped back to work on other things, like managing the Discord server and building simple miscellaneous Discord bot tools, which introduced me to Node, Heroku and Git. These are still active now.

Abyo

As the months passed and I monitored user feedback, I became unhappy with aspects of the new Pyobot’s operation:

• The app was structured as a cascade of asynchronous operations – poll map, parse data, filter it, send it – that had no priority system or control of when they finished executing. Public channels bring the most user utility and so should’ve worked regardless of load from the (by-now hundreds of) users of the private service. Also, calls for new data occurring before the current data had been processed were leading to exploding lag and late posts – for a time-critical notification service – to the extent that the bot was unable to poll all available data from the map. The map’s own design flaws meant that all data had to be polled for certain channels (e.g. 100% IV) to work, so their functionality was seriously impaired.
• Implementing unusual filters was hard. Pyobot’s filter design is mostly the same as Percy’s one, but user ideas for new channels were often stretching what was possible and required hardcoding each time.
• Running the bot was not collaborative. If there was a new channel to implement or a crash, we had to wait for one person to do it or fix it.
• I had ideas for how to make the polling faster and design the notification format better.

In the meantime, I had read a lot about programming concepts like asynchronicity in JavaScript, as well as classes and scoping, and had more experience with testing, so I was confident I was up to redesigning the public part of Pyobot, as my first complete non-improvised project. Also, somebody had volunteered a server with remote desktop access for the project. I began in November 2017.

First, I wrote locate.js. I had the idea that instead of waiting for internet APIs to tag alerts with geographical information (postcode, suburb, etc.), I should download data, process it, and use geometric algorithms to tag spawns entirely locally. Moriakaice had already done this with station data and a closest point algorithm; I extended this to suburbs (modifying the data to resolve its inaccuracy, which had bothered me since my original version of Pyobot), and found GeoJSON polygon data for postcodes and boroughs, and a point-in-polygon algorithm from the Node package Turf.js.

I released locate.js separately (with data) at first. Moriakaice implemented it in Pyobot, and the speed improvement was such that suddenly, all available data could be polled (so the 100% IV channel worked), and users of the private service could thus track whatever they wanted. Which was unexpected.

So, even though this module inadvertently resolved the biggest problem with Pyobot, I persevered with my project. These were the other improvements I made:

• Abstract filters. I redesigned the filter syntax for the third time. This time, filtering is handled by the “Agent” class, whose objects contain a dictionary of Discord channel IDs, and a filter pure function that maps Pokemon objects to a list of keys of the dictionary, according to where that Pokemon should be sent. These two things are read from files representing each Agent in an agents/ folder, meaning that anyone with remote desktop access to the bot server can design filters, and implement them simply by saving to the folder and restarting the app. A step towards a collaborative bot.
• Faster polling. All previous versions polled at fixed intervals (the source updates approximately every 30s). Abyo polls every 29s, and if the source reports a last-updated time equalling what was received on the last poll, it tries again in 2s. This way, it tracks the source’s update time without gratuitous pinging and gets the alerts out faster.
• Program control flow and error handling (The latter compared to my previous Pyobot version). Abyo synchronously polls, parses and filters all information, and only then calls the next iteration of this loop with timeout as per the polling logic. By watching the console, it is immediately clear how fast and well it’s keeping on top of new data. All send operations are asynchronously called by the Agents. They don’t report back to the main loop (I may decide they should in future) but they do report their own success or failure to the console.
• Notification design. I had a much better idea of what users wanted from their notifications as a result of watching Pyobot in use for a few months, so I redesigned the notifications, using line breaks, testing using mocked iOS/Android fonts, etc.

And the bot has been working, faster, more user-friendly and more reliable, ever since! With v1.1, I introduced a couple of cool new features:

• A major (i.e. 15× faster) optimisation to the point-location-via-point-in-polygon algorithm, through skipping unnecessary PiP tests. If you haven’t already read what it is in the changelog on the GitHub repository, I won’t spoil it as I like to set it as an exercise to people I chat to to see what ideas they come up with. The result is that I’m confident that if people draw polygons with geojson.io, I can use them in the actual filters, to serve public channels on more local Discord servers with area-based filtering!
• A conceptual distinction between pre- and post-processing spawns, and where the relevant logic is written.
  • Pre-processing tags every spawn polled from the map with information that can be used to filter by.
  • Post-processing tags every already-filtered spawn with extra information for the notification, and is only done to spawns that some filter has accepted, so is allowed to take much longer per spawn.

As a result of my PiP optimisation (above), I was happy to move postcodes from post- to pre-processing. Hence, a local Discord server could simply specify a postcode to determine the region it filters for, to avoid designing a polygon.

On the coding side, I learned a lot of things:

• Proper control flow with clean syntax wouldn’t have been possible without JavaScript’s promises. I designed the synchronous loop with async/await and tail-recursion via setTimeout, and the asynchronous sending (and every HTTP operation) with promises that usually resolve with a console log/error message.
• My handing of Pokemon and Agent objects led to my first designs of classes. Agents have custom properties and methods (e.g. filter functions) imported from files, as well as standard methods (e.g. sending procedure) attached to their prototypes.
• I’ve considered testing much more actively during the project. While I wasn’t brave enough to go for unit test coverage, I did write tests with mock I/O (e.g. fetch and save data, or read saved data and process it, mock-sending to console), and tested each function via Node’s REPL. I also prepared some test Pokemon and Agent objects.
• I learnt some basic concepts from computational geometry, and geographical tools and conventions, in thinking about how to do location tagging and filtering.

And there are still improvements that could be made:

• Error information should be dumped so it doesn’t get lost after hours of program operation. v1.2 EDIT: done!
• I could install a Geographical Information System program to help me fix deficiencies in the geo data (like postcode boundaries catching opposite banks of the river). v1.2 EDIT: done!
• I have been thinking about different ways of implementing concurrency – what to parallelise, how to use threads, processes, whether to switch to GoLang, etc. I don’t see an obvious choice between the many models of concurrency I’ve thought of for this project, so haven’t tried to implement one. Besides, it’s purely of interest to answer the question of arbitrary scalability; there won’t be a practical case for it for a very long time, if ever. v1.2 EDIT: still thinking!

And that’s the story so far!