grand exchange api [Free] humblePhlipper: Grand Exchange flipping with automatic item selection (300+ gp/hr, 0 xp/hr)

[mirko2000 0]

But doesn't the Osrs wiki API updates every 5 min so the value will always be the same withing the 5 min time frame

[apnasus 49]

1 hour ago, mirko2000 said:

But doesn't the Osrs wiki API updates every 5 min so the value will always be the same withing the 5 min time frame

In a word, no. That's what makes this new API released ~2 years ago special, it includes continuously updated latest prices. (I believe the OSRS Wiki prices site includes a 60 second refresh option.)

There is a timeseries route with 5m, 1h, 6h, and 24h timesteps for average prices, but I use the latest route for pricing at the moment.

As of V1.20, humblePhlipper displays latest bid/ask prices on the Paint display and you should see these change more frequently than every 5 minutes for most medium-high volume items (e.g. Logs and Death rune).

[thevance102 2]

Mmm you could make this bot way better, it should only buy 1 at a time to gain margins, bulk selling to find out margins means you're just gonna loss unessecery money over time

1.buy 1 amethyst arrows for 380 (overpriced)

2.selling for 1gp (to find out margin)

3.Sell for that amount

instead it just bulks sells meaning u can potentially lose lots of money.

[apnasus 49]

53 minutes ago, thevance102 said:

Mmm you could make this bot way better, it should only buy 1 at a time to gain margins, bulk selling to find out margins means you're just gonna loss unessecery money over time

1.buy 1 amethyst arrows for 380 (overpriced)

2.selling for 1gp (to find out margin)

3.Sell for that amount

instead it just bulks sells meaning u can potentially lose lots of money.

This is actually a great way to uncover bid/ask prices hidden in the limity order book which I've seen youtubers do but hadn't thought to implement. I do see a few problems for implementing it in a bot though:

* This works for high volume items, like Logs and Death rune, but for low volume items with non-trivial costs (e.g. Bear feet) there's a chance you really would sell at 1gp, whereas the upper limit for discovering asks is 2,147,483,647 - so you could lose a lot of money. (Of course a human wouldn't, but this bot isn't that sophisticated yet).

* For the aforementioned lower volume items, you probably aren't discovering bids/asks which you could reasonably expect to replicate in a short time frame. I might get 1gp for my Bear feet where someone's put in a stupidly low overnight offer, but if I bid at 1 or 2gp I'm unlikely to make a buy in the next 4 hours because no one's likely to try sell their Bear feet that low.

* For the above two reasons, this method works better with medium-high volume items. I could therefore just implement this method for the medium-high volume items, but then I'd have to decide what those were, a problem in itself. More likely I'd offer this pricing algorithm and let you guys chose it if you wanted to.

* If I sell the Amethyst crystal at 1gp and it executes for 340gp, I could try bulk buying it at 340/341gp. But then there is no guarantee that someone won't come in, without me knowing, and bid at at 345gp. I would know someone was bidding higher than me if I checked the latest API and saw that since I entered my order, the latest bid price had jumped to 345gp. I could then raise my bid price 345gp, but then we are back to square one and using latest realised bid/ask API prices again instead of your discovery mechanism. To rectify this, we could recurrently use your method - do it once every 60 seconds for example. But even once every 60 seconds and even for a bot, at 8 traded items that is going to significantly slow down time spent offering trades and cause you to constantly lose your position in the limit order book (older orders get served first). Further more 60 seconds isn't a very frequent update at all, with 8 items v1.2 currently updates every 1.0 seconds if there's no cancelling/collecting/asking/bidding happening.

* If selling and you have already hit the buy limit, you cannot buy one more of the item (at some absurdly high price) to discover the current ask.

For the above reasons, and in particular the penultimate one, I think this is a method probably better suited to humans than to bots. It's probably most useful to people who don't like using price checkers or for when price checkers have unusually stale prices.

Finally, I would add that humblePhlipper doesn't buy or sell to discover ask/bid prices. If it's bulk selling at a loss, it's not to try 'test the market' and discover its next entry price, it's because the ask price has fallen below the price it paid and it's simply following the latest ask - offloading inventory at the latest ask is its first and foremost rule. There's no guarantee ask prices will rise again, although they usually do since in my experience losses are usually caused by the bot naively mimicing mispriced bids and asks, but also by always selling inventory it keeps the bot nice and 'clean' and makes it easier to quickly shut down by clicking pause+resume with a healthy net zero item position and end-of session pure profit logging.

53 minutes ago, thevance102 said:

instead it just bulks sells meaning u can potentially lose lots of money.

I did discover this the hard way fairly early on which is why I introduced the maxBidVol parameter. It is no means perfect, but it does limit the magnitude of the loss and means the bot stops trading the item until margins are profitable again. Setting maxBidVol at 0 means it will never buy anything, setting it at 100 means it will always try buy the 4hr GE limit, you are free to set it anywhere in between - at the moment I usually go with 20-40% on a 2m to 10m cash stack.

Edited December 23, 2023 by apnasus
grammar

[tmanowen 8]

Loving the amount of effort that was put into this script. This is the first public flipping script I've seen on any platform in years.

 

While I do like the 5m(or 60 second) update method, and I do agree it's more effective than the buy 1 sell one method based on how the script runs, I just don't know if the script is properly utilizing it. From my testing with it, it is refreshing on the paint, and with new offers, but isn't reflecting on already existing offers. So for example, if an offer is left in at 100k, and then the 5m/60s refresh comes through to sell at 95k, it doesn't cancel that offer. Rather, it is waiting for it to buy at 100k, to then sell it at 95k. I'm not sure if this is readable information and then can be implemented, but if it's possible, I think the script would be unstoppable. Running it for ~4ish hours, that was the biggest issue I noticed. Other than that, sometimes it was taking orders out too quickly and just putting them back in instantly, I don't know what was causing that but just seemed almost random. Thanks for sharing this and I'll be sure to post anything more I notice about the script. 

Here is my current proggie, starting with 4m on members:

 

Edit: 

Ran this again today in a longer setting and didn't look like it performed as well as yesterday unfortunately. Below is the proggie:

I'm going to make some changes to the items (taking away the ones that lost me money and leaving the other ones) and run it overnight and see if it's still not banned and profiting when I wake up. 

Edit: 
Had a decent run last night but my PC went to sleep and didn't get an accurate proggie. It made 400-600k. I ran it today for a bit and here's my proggie with the edits in place

Edited December 25, 2023 by tmanowen
edited for proggies and updated info

[apnasus 49]

On 12/24/2023 at 6:37 AM, tmanowen said:

Loving the amount of effort that was put into this script. This is the first public flipping script I've seen on any platform in years...

Thanks so much for all the feedback and especially the proggies! To address the issues you raised, let me quickly outline the mechanics in more detail, V1.2 is a recurrent onLoop():

(1) Update API prices

(2) For each slot, check cancel condition: if(offer price != API price OR offer is bid and spread not profitable OR offer is bid and partially complete) { cancel; }

(3) For each slot, check collection condition: if(slot is done trading, i.e. green or red bar due to fully complete or cancellation) { collect and log; }

(4) For each item in selections, make ask: if(we have inventory) { makeSellOffer; }

(5) For each item in selections, make bid if(spread is profitable) { makeBuyOffer; }

-> Return to (1)

So API price should update every ~1 second if nothing is happening or every ~4-5 seconds if there's lots happening in (2)-(5)

On 12/24/2023 at 6:37 AM, tmanowen said:

So for example, if an offer is left in at 100k, and then the 5m/60s refresh comes through to sell at 95k, it doesn't cancel that offer. Rather, it is waiting for it to buy at 100k, to then sell it at 95k.

If I understand you correctly, bid was at 100k and the ask was 105k so it bids at 100k. There is a buy offer at 100k and the Ask price in the Paint updates to 95k, but it is not cancelling the buy offer. This is possible if the item is in, say, the 8th slot. The script updates prices in (1) (and then updates Paint), and whilst checking the other 7 slots for cancel in (2), the buy goes through at 100k before it can get to the 8th slot and cancel it. Since the script always sells at current Ask, it will then sell at a loss. If the Ask Price drops to 95k and the buy offer stays up at 100k for 20+ seconds or several minutes, then that is most certainly a bug. Is that what's happening? Any more insight or screenshots you can share here would be greatly appreciated.

On 12/24/2023 at 6:37 AM, tmanowen said:

Other than that, sometimes it was taking orders out too quickly and just putting them back in instantly, I don't know what was causing that but just seemed almost random.

This is an issue I'm aware of and have seen myself. What's happening here is that the OSRS wiki live price API sometimes accidentally serves old data (i.e. the latest bid/ask price just before the actual latest bid/ask price) so it will cancel the offers, but then the next loop the API serves the correct/current data so it just puts the offers back in. I hadn't got round to this yet but thanks for bringing it to my attention, I just need to cache the latest bid/ask times attached to API JSON data and compare these against incomming data to decide whether to update prices with the incomming API data.

I haven't been on the forums in a couple days because of Christmas, but in the next ~24 hours or so I should be requesting a compile for V2.0 with new features:

* Threaded API updating prices every 1.0 seconds regardless of onLoop() status

* Custom bid order priority (which item to place bids on first for your selected items) based on profit margin, 1hr volume, capital binding, or some weighted combination of the three (in V1.20 bid priority is randomised every loop) (shoutout to @demstar  for the suggestion here)

* Automatic item selection using filters like those on price checkers (volume above 1,000,000, profit margin above 2, etc.)

* Custom pricing (at the moment, options will be Latest, 5 Min Avg, 1 Hr Avg, Best_of{Latest, 5 Min Avg}, in the future I will add more sophiticated pricing algorithms, I think I'll start with Kalman Filters).

* Scrapping the target volume parameter in favour of locally saved used limits (i.e., JSON files keeping track of how much of an item you've bought, on a given account, and when you're 4hr GE limit refreshes to automatically calculate how much of an item you can still buy).

* Improved trade loggin as on one occassion V1.20 failed to log a sale correctly (logged a price of zero) which skewed calculations.

* Restart onLoop() from the beginning each time there is a successful cancellation/collection/bid/ask instead of continuing with the loop (this will mitigate losses when starting an 8-slot session where purchases are made, e.g. on slot 1, and then Ask falls whilst you are still making bids on slot 2 through 8).

* Far more custom config profiles

Here is a preliminary sneak peak of the GUI for V2.0,

API threading and restarting onLoop() after a succesful interaction might fix the first issue you described but I'm not 100% sure, I will be doing more extensive testing in the days to come. If you have any other suggestions or parameters you'd like to see in V2.0 please do let me know!

On 12/24/2023 at 6:37 AM, tmanowen said:

Finally, for this one, I think you could have mitigated some of the big losses by setting max bid order at less than 100% (in V2.0 this parameter will be replaced by absolute Max Bid Value and Max Bid Volume.) I'd also love to hear what items you're trading or even see some of your CSVs (I've started a github of CSVs which I'll be making public fairly soon, and might introduce the option to save CSVs locally for V2.0). It's also cool to see the script having been run for a 9 hour session!

Sorry for the essay, hope you're having a happy holiday!

[tmanowen 8]

23 hours ago, apnasus said:

(1) Update API prices

(2) For each slot, check cancel condition: if(offer price != API price OR offer is bid and spread not profitable OR offer is bid and partially complete) { cancel; }

(3) For each slot, check collection condition: if(slot is done trading, i.e. green or red bar due to fully complete or cancellation) { collect and log; }

(4) For each item in selections, make ask: if(we have inventory) { makeSellOffer; }

(5) For each item in selections, make bid if(spread is profitable) { makeBuyOffer; }

-> Return to (1)

So API price should update every ~1 second if nothing is happening or every ~4-5 seconds if there's lots happening in (2)-(5)

Really cool to understand how the script actually works. Will take this now into consideration when I choose items / watch results. Appreciate it 

23 hours ago, apnasus said:

If the Ask Price drops to 95k and the buy offer stays up at 100k for 20+ seconds or several minutes, then that is most certainly a bug. Is that what's happening? Any more insight or screenshots you can share here would be greatly appreciated.

Yes. This is what was happening when I mentioned that in one of my edits. I haven't seen it firsthand happen since so maybe it was a fluke / one-off bug occurrence. 

On 12/26/2023 at 1:13 PM, apnasus said:

I haven't been on the forums in a couple days because of Christmas, but in the next ~24 hours or so I should be requesting a compile for V2.0 with new features:

* Threaded API updating prices every 1.0 seconds regardless of onLoop() status

* Custom bid order priority (which item to place bids on first for your selected items) based on profit margin, 1hr volume, capital binding, or some weighted combination of the three (in V1.20 bid priority is randomised every loop) (shoutout to @demstar  for the suggestion here)

* Automatic item selection using filters like those on price checkers (volume above 1,000,000, profit margin above 2, etc.)

* Custom pricing (at the moment, options will be Latest, 5 Min Avg, 1 Hr Avg, Best_of{Latest, 5 Min Avg}, in the future I will add more sophiticated pricing algorithms, I think I'll start with Kalman Filters).

* Scrapping the target volume parameter in favour of locally saved used limits (i.e., JSON files keeping track of how much of an item you've bought, on a given account, and when you're 4hr GE limit refreshes to automatically calculate how much of an item you can still buy).

* Improved trade loggin as on one occassion V1.20 failed to log a sale correctly (logged a price of zero) which skewed calculations.

* Restart onLoop() from the beginning each time there is a successful cancellation/collection/bid/ask instead of continuing with the loop (this will mitigate losses when starting an 8-slot session where purchases are made, e.g. on slot 1, and then Ask falls whilst you are still making bids on slot 2 through 8).

* Far more custom config profiles

Here is a preliminary sneak peak of the GUI for V2.0,

Thanks for sharing this! I'm excited to see check out and mess around with all of the new features. 

On 12/26/2023 at 1:13 PM, apnasus said:

Finally, for this one, I think you could have mitigated some of the big losses by setting max bid order at less than 100% (in V2.0 this parameter will be replaced by absolute Max Bid Value and Max Bid Volume.) I'd also love to hear what items you're trading or even see some of your CSVs (I've started a github of CSVs which I'll be making public fairly soon, and might introduce the option to save CSVs locally for V2.0). It's also cool to see the script having been run for a 9 hour session!

Sorry for the essay, hope you're having a happy holiday!

I don't quite understand the Max bid 100% function, probably a waste to explain to me now so with the new versions release, would you care to explain the Max Bid Value and Max Bid Volume functions that you are planning on creating? 

I'm currently running a small list of near guaranteed profits (or recoverable profit) items that even if they go down, they will stay up overall / throughout time on. Still need to make some changes to my list and could add more to make it more profitable.

I'm not super familiar with Dreambot, so if you can share some step by step on how I can share my CSV's with you, I'd happily do so. I haven't showcased publicly in my proggies the items / profits directly because if someone were to throw in my item list in a mass scale, my bot likely wouldn't do as well. 

Hope you had some Happy Holidays as well mate. I'm still flabbergasted that someone has gotten one of these to work so well, the last one I used, which was maybe around ~2017, wasn't even close to as advanced as your current version. 

[apnasus 49]

2 hours ago, tmanowen said:

I don't quite understand the Max bid 100% function, probably a waste to explain to me now so with the new versions release, would you care to explain the Max Bid Value and Max Bid Volume functions that you are planning on creating?

For V1.20 you get to chose your 'target volume', the amount of an item the script will try buy and sell. By default this is the 4hr GE limit. The maxBuyVol parameter is a % between 0 and 100 determining the maximum quantity to try buy at any one time. If the target volume was 18k and maxBuyVol was 20%, it would only put in buy offers of 3.6k quantity at a time. I introduced this to limit losses when spreads become unprofitable, it will take a loss on that first 3.6k but then it won't buy more as the spread is unprofitable.

For V2.0 I've scrapped the 'target volume' parameter, which is now automatically set to your remaining GE buy limit, as V2.0 automatically logs/checks your 4hr limits and 4hr limit refresh times, so `target volume` is always set to whatever your remaining limit is. `maxBidValue` will set the maximum amount of money you're willing to tie up in a slot at any one time, `maxBidVol` will set the maximum amount of any item you're willing to try buy at any one time.

2 hours ago, tmanowen said:

I'm not super familiar with Dreambot, so if you can share some step by step on how I can share my CSV's with you, I'd happily do so.

With DreamBot and the script running, if you click the client at the bottom

i.e. where it says 'Starting humblePhlipper now!', it will bring up the console log. The script logs every trade enscapsulated by <trade></trade> and at the end of the session it logs a CSV encapsulated by <trades></trades>

In the top/first post of this thread there's an example of how to access and manipulate this in Python for analysis. My 'test runs' have further python examples for how you can analyse this data. For V2.0 I plane to introduce the option for you guys to save these CSVs locally automatically.

Still wriggling out some of the details for V2.0 and got caught up in IRL but will be releasing extremely soon, here's some more pics. Auto-selection targetting profit margin:

Auto-selection targetting volume,

The order for the above selections is set in `Parmas` under `Bid Priority`, which also continually orders selections as the trading session goes on to prioritise which items to try and buy,

Thanks for all your positive words!

[tmanowen 8]

14 hours ago, apnasus said:

For V1.20 you get to chose your 'target volume', the amount of an item the script will try buy and sell. By default this is the 4hr GE limit. The maxBuyVol parameter is a % between 0 and 100 determining the maximum quantity to try buy at any one time. If the target volume was 18k and maxBuyVol was 20%, it would only put in buy offers of 3.6k quantity at a time. I introduced this to limit losses when spreads become unprofitable, it will take a loss on that first 3.6k but then it won't buy more as the spread is unprofitable.

For V2.0 I've scrapped the 'target volume' parameter, which is now automatically set to your remaining GE buy limit, as V2.0 automatically logs/checks your 4hr limits and 4hr limit refresh times, so `target volume` is always set to whatever your remaining limit is. `maxBidValue` will set the maximum amount of money you're willing to tie up in a slot at any one time, `maxBidVol` will set the maximum amount of any item you're willing to try buy at any one time.

Ahhhh ok I get it now thank you! This clears it up. I feel like I might miss the maxBuyVol feature now that I understand it but realistically the new v2.0 features will just make it more consistent and exact for specific items to be set one way, and others set other.

14 hours ago, apnasus said:

With DreamBot and the script running, if you click the client at the bottom

i.e. where it says 'Starting humblePhlipper now!', it will bring up the console log. The script logs every trade enscapsulated by <trade></trade> and at the end of the session it logs a CSV encapsulated by <trades></trades>

Thanks for all your positive words!

I didn't realize I could only do it after a session, I guess that's what your locally saved CSV feature will fix. The account I was testing this just ran out of members so once I get a new Prime I'll run some long proggie tests again and try to share this to you. Really eager to check out all of the new features in v2.0 and increase my profits with it. Happy New Year. 

[williambruhn1 1]

im kinda new to this stuff and dont know if its easy or possible for you to maybe put in more flipping items too choose from the list without having to know the ID and put them in manually, i would really appriciate if that would be possible, otherwise great script that works flawless