Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Many banks, hedge funds use kdb+/q for time series databases. This (very expensive) software is literally unheard of outside of these niche domains. I've been using it for close to 5 years for high frequency data, and honestly nothing out there comes close to this awesomeness of kdb+


I would be interested to hear what you consider high volume (writes/second). I am supporting a manufacturing system and it sits at the moment at 7 000 writes/second (it is a normal time series, ie id,time,value,quality).


7000 writes per-second is pretty low for many high-volume time series needs.

Though it's not usually write throughput that most of these technologies are worried about. It's usually compression using dsp methods, aggregate stream folding computations, etc... that matters.


Still waiting what is considered as high. Compression etc are not part of the question.


Just queried my db, and 300k a second is a typical peak in CME futures (across all products). I'd imagine that options could be a lot more than that.


If I may ask, what are you using as db?


kdb


I worked in adserving. On peak moments we were doing around 350,000 updates a second.


Chill dude, I was being conversive. Your attitude makes me not even want to provide you with my own data now.


That took some restrain. Thanks for being nice. I got your point originally but some people think everything needs to be an argument or worse a fight.


Engineers wanting data isn't exactly surprising. I would not consider it attitude.


The systems I deal with start at millions of writes per second and go up from there. I have heard of systems that do over a billion writes per second, though I have not breached that threshold personally (yet).

From an IoT or sensor network standpoint, 7000 writes/second is an idle server.


what db are you using?


I don't have numbers handy. The real power of kdb+/q comes from the column oriented architecture and the extremely powerful vector functional language q. The language q is inspired by APL. I highly recommend you to check out this article to get a sense of the APL family of languages https://scottlocklin.wordpress.com/2013/07/28/ruins-of-forgo...

If you want a database for blazing fast data-storage and retrieval, there are many options available. You start seeing the real benefits of kdb+/q when you use q to simplify very complex operations that aren't easily done in SQL. Also, the high level operators that q makes your code extremely terse. I've written complex backtesting systems that perform data mining on massive datasets - all in one page of very tight q code!


Are you using SQL or ?


ISAM


How much would the price be?


There is a free 32bit version of kdb available (http://kx.com/software-download.php). For the commercial version, pricing information is not publically available.


The zip files on that page contain the source for kdb, I thought I'd take a look at how it works.... nope! That is impenetrable


It's impenetrable to you, in the same way Mandarin is impenetrable at first glance to someone used to reading Latin languages.


The thing with KDB is almost all uses of it are in memory deployments. It isn't hard to make something that has little persistence or relegates persistence to a 2nd class citizen to run quickly.


Well, but it does not exists as open source.


no, but std::map does


Certainly, but it does not provide the same benefits as the complete package.


Not even J?




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: