Page 1 of 2 12 LastLast
Results 1 to 10 of 20

Thread: Hexsales API

  1. #1

    Hexsales API

    Hey guys,

    I'd like to release my hexsales.net api for public use. It let's you query for some basic statistical data of sales from Hex' economy. This thread shall be used as an official thread for this side-project.

    It is available over here. It's documentation is sorta complete, but obviously there will be some questions so please let me know. I guess the 'data model' part needs some more clarification.

    I think this way I can concentrate on what I like the most (backend stuff) and can stop working on rest clients (aka hexsales.net as a website), which I never really liked. Besides, I'd really like to see some integration of this in some 3rd party community projects/applications.
    Hexsales - Visualizing auction house data (currently offline)
    Hexsales API
    HEX card/item/... database api (Code/Docs on GitHub - queriable database api for cards, equipment, etc.

    Email: cwik@hexsales.net

  2. #2
    Nerfed Baby Yeti AA
    Join Date
    Jun 2014
    Location
    Throne room next to a Round Table
    Posts
    1,701
    Api-ception :O

  3. #3
    Yeah I know^^. Believe it or not I was planning that for a while.
    Hexsales - Visualizing auction house data (currently offline)
    Hexsales API
    HEX card/item/... database api (Code/Docs on GitHub - queriable database api for cards, equipment, etc.

    Email: cwik@hexsales.net

  4. #4
    This is pretty exciting I was making a small project and this'll make it a lot easier, thanks

  5. #5
    Needless to say I'd like to hear feedback and ideas from you guys. I can also share some information about how I create the data I serve, etc.
    Hexsales - Visualizing auction house data (currently offline)
    Hexsales API
    HEX card/item/... database api (Code/Docs on GitHub - queriable database api for cards, equipment, etc.

    Email: cwik@hexsales.net

  6. #6
    /histories now has an additional, optional parameter step, which allows you to specify a dimension in which the data will be grouped. For more info see the documentation.
    Hexsales - Visualizing auction house data (currently offline)
    Hexsales API
    HEX card/item/... database api (Code/Docs on GitHub - queriable database api for cards, equipment, etc.

    Email: cwik@hexsales.net

  7. #7
    Started fiddling around with this. Great stuff.

    I'm probably going to have some questions later, but this is very nice.

    Minor feedback:
    - "amount" was a bit confusing at first glance as it is often used to represent monetary value, "quantity" would be more clear imho
    - Median for history data would be very useful, if feasible to implement. Average is much less useful because there is so much price volatility.

  8. #8
    Hey,

    thank you!

    1. I guess you are right. I'll change that in the near future.

    2. This is a bit more complicated. My database actually holds already aggregated data. For every unit (card, pack, piece of equipment, etc) and currency there is zero to one entry per day. Every entry holds aggregated data like min, max, median, average. Before I started using this data model, I had saved every single sale, which made it impossible to calculate median values over.

    I'm currently not using the median values for statistics nor histories as I'm not sure how to use those values in my sql aggregations. My statistics endpoint basically aggregates over the whole data while histories allow you to specify a size how much data should be combined (step parameter). To come to my point, I can either aggregate an average of daily median values or a median of daily median values. In both cases I'm not entirely sure as to how useful that piece of information is.

    Besides I'm not even sure if I want to deal with medians on database level ever again, as it simply does not scale in the long run. As soon as set 4 hits, there probably is no need for median values anyway. But providing an average of daily median values should be quite easy and somewhat proficient.
    Hexsales - Visualizing auction house data (currently offline)
    Hexsales API
    HEX card/item/... database api (Code/Docs on GitHub - queriable database api for cards, equipment, etc.

    Email: cwik@hexsales.net

  9. #9
    The Transcended
    Join Date
    Jun 2013
    Location
    California
    Posts
    7,860
    Quote Originally Posted by cwik View Post
    2. This is a bit more complicated. My database actually holds already aggregated data. For every unit (card, pack, piece of equipment, etc) and currency there is zero to one entry per day. Every entry holds aggregated data like min, max, median, average. Before I started using this data model, I had saved every single sale, which made it impossible to calculate median values over.

    I'm currently not using the median values for statistics nor histories as I'm not sure how to use those values in my sql aggregations. My statistics endpoint basically aggregates over the whole data while histories allow you to specify a size how much data should be combined (step parameter). To come to my point, I can either aggregate an average of daily median values or a median of daily median values. In both cases I'm not entirely sure as to how useful that piece of information is.

    Besides I'm not even sure if I want to deal with medians on database level ever again, as it simply does not scale in the long run. As soon as set 4 hits, there probably is no need for median values anyway. But providing an average of daily median values should be quite easy and somewhat proficient.
    I would expect the DB to hold the raw data. Isn't storage cheap these days? Summary data can be stored in a data warehouse that is updated periodically (once per day).

    Also, I believe there are ways to maintain a median while only updating it incrementally, but it won't have perfect accuracy. Perhaps try this:
    http://web.ipac.caltech.edu/staff/fm...s/Remedian.pdf

  10. #10
    Quote Originally Posted by Yoss View Post
    I would expect the DB to hold the raw data. Isn't storage cheap these days? Summary data can be stored in a data warehouse that is updated periodically (once per day).

    Also, I believe there are ways to maintain a median while only updating it incrementally, but it won't have perfect accuracy. Perhaps try this:
    http://web.ipac.caltech.edu/staff/fm...s/Remedian.pdf
    It's not about storage. It's about quickly aggregating about time-series data. Calculating sums, minimums/maximums or averages doesn't take that long, but calculating median values over tens of thousands to millions of records (which is the use-case for my /statistics and /histories endpoints) takes way too long to calculate as most algorithms and aggregate functions have to sort all values in memory before determining the result. The sad thing is that there is nearly no optimizing to be done, as all rows have to be scanned.

    So what I did is pre-aggregate the daily data. This way my queries execute 5-15x faster than with storing all sales seperately. Furthermore I'm not even losing that much accuracy of my data. If I want to get my basic 'statistical' set of data, like min, max, sum and average, I can dig that up by manually determining the average by dividing the sum of 'daily_sums' by the sum of 'daily_amounts' or quantities. With min, max and sum I can even use min of 'dailymins', max of 'dailymax' and sum of 'dailysums'.

    So the only stat that I cannot fully rebuild from my pre-aggregated data is the median value, which should only be interesting for very slim time-frames, like checking the daily history data for AA versions of swingy cards. As soon as you select weekly or even monthly data, usability of a median value should decrease drastically. If you'd still insist on it, I can easily provide you with an average of the daily median values. That way, selecting daily history data stays untouched and you get the 100% correct value, but as soon as you select weekly or above, the accuracy of the 'avg(median)' falls of as does its use-case.
    For /statistics I think the whole a median value isn't really needed, as it should be used as a tool to find basic summarized data for either:
    1. a large timeframe but a small quantity of cards, equip, etc (eg. monthly data for Vampire King)
    2. a short timeframe but a large quantity of ... (eg. daily data of all legendary or set 3 cards)
    Hexsales - Visualizing auction house data (currently offline)
    Hexsales API
    HEX card/item/... database api (Code/Docs on GitHub - queriable database api for cards, equipment, etc.

    Email: cwik@hexsales.net

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •