Sphinx 0.9.10 teaser

We've been pretty busy lately on a number of cool new features (again), and an alpha will take some more time, so I thought it's about time for a sneak peak into 0.9.10, as well as few other things.

The most major change is unarguably string attributes support. Strings are intentionally stored in RAM, for speed. On-disk blobs are considered in the roadmap but not yet planned. Currently strings can only be used for storage, ie. you can fetch them, but not yet sort nor group on them. However both sorting and grouping is perfectly technically possible, and will be implemented at some point (enters SET NAMES).

Second-best one is in my opinion common subtree cache. It lets searchd identify common query subparts and cache them within a query batch. So if you're searching for "john connor rocks" and "john connor sucks" in the same single query batch, searchd will only search for "john connor" once, and reuse the partial result. The feature's currently pre-alpha, but preliminary testing already shows 2x-4x improvement in some production scenarios.

Other forthcoming features include index precaching progress on searchd startup, some work on rankers, query-based snippets generation, new "blended" character class that gets indexed both as a valid character and as a separator (think of & in AT&T); support for hitless indexes, etc. So it's going to be cool new alpha again.

As of 0.9.9 stable branch, a bunch of bugfixes was committed since 0.9.9-rc2 release, currently available via Google Code public SVN mirror. No super-major affect-everyone bugs had been spotted yet, so the plan's to give it another round of fixing in a couple of weeks and then declare it release.

Downloads and Powered page are redesigned now, should be more usable. Everyone who submitted the sites for Powered in the recent months, I'm sorry it's taking so very long to process your submissions, but unfortunately we seem to have way too much stuff on our hands and just not enough hands to handle it all. Your submissions are not lost forever and will be posted.

Last but not least, Sphinx made it into the second and final phase of Sourceforge Awards 2009, in SysAdmins and Enterprise categories. (We almost made it in Developers too.. but with an "almost", alas.) Voting ends by July 20th, so please vote for us, bring all your friends, and spread the link! :-)

Permalink // del.icio.us // digg // stumble

#1. Pete Spicer | 2011-05-03 13:37:05

String support is going to be huge. I seem to recall mentioning this was one of the big features that I considered necessary for 1.0 a while ago (though in the time since I realised that there aren't so many uses cases that can't be worked around)

0.9.10 is going to be pretty awesome.

#2. cysin | 2009-07-13 03:04:08, anonymous

Sounds great, thank you~

#3. Floren Munteanu | 2009-07-13 03:24:25, anonymous

As usual, more awesomeness from Andrew. Thanks for your dedication and congrats for being a Sourceforge finalist. All I wish you now is to get a bunch of large clients who will pay your bills. My small donations are not enough, unless others join my initiative.

#4. Tequila | 2009-07-23 03:11:36, anonymous

String attributes will be support!!! That's great message, i'm glad to see it.

#5. Jeremy Wei | 2009-07-28 14:50:10, anonymous

"String support" that is i need. hope the release.

#6. Pratik Khadloya | 2009-08-04 04:56:10, anonymous

Yep! String attributes support is needed, it will make the polymorphic relational search work.

#7. �ukasz Korzybski | 2009-08-04 08:43:05, anonymous

Great news! I realized that I badly need string attribute feature while I was implementing dynamic category tree in online shop application.

To generate this tree I used MVA attr for holding list of categories per each article that was in a search result. The problem is that MVA attr values are sorted by Sphinx so my category tree was all messed up.

#8. Maxim McNair | 2009-08-04 19:15:38, anonymous

What do you mean by polymorphic relational search?
Can you give me a practical application?

If you were using MVA to specify in which category an article was I cannot see how string attributes will help you.

#9. �ukasz Korzybski | 2009-08-05 13:44:16, anonymous

to Maxim:

I'm using MVA to build dynamic category tree which shows in which extact categories the search results are.

Each article has list of categories starting from root category and up to the category which directly contains this article (going deeper into the tree).

Having such a list for all articles in the search result, I can build category tree which will be displayed along with the search result.

Using MVA for this purpouse is problematic casue MVA lists are sorted, so category list (per article) gets messed up, cause parents and children are being disordered.

Currently, as a hotfix, I add <category level>* 1 000 000 to each id on the list, so sorting doesn't affect ids.

#10. �� | 2009-08-11 16:11:38, anonymous

��,�Һ�ϲ��

#11. Raj | 2009-08-12 03:23:57, anonymous

Need realtime search.

#12. clinton | 2009-08-13 09:53:48, anonymous

stored string when sphinx doesnt support stored original source is just dumb. unless the string is tokenize or static (label) and search in similar to dictionary ways.

#13. chinafacttours | 2009-08-21 05:56:40, anonymous

It seems a good idea! Does it work well for a travel site?

#14. ethaniel | 2009-08-24 19:45:10, anonymous

NEED REAL TIME UPDATES! My god, Andrew, you've been promising this for 2 years already. How much more should we wait?

#15. Ythan | 2009-08-27 15:54:31, anonymous

Wow, that's quite the sense of entitlement you've got there ethaniel. Let me ask, how much have YOU paid to sponsor this feature's development? And if it's so essential to your application, why have you waited two years instead of pursuing other alternatives?

#16. etre | 2009-09-03 03:31:40, anonymous

thanks,"String support" that is i need,
i thanks,Strings are stored in RAM or on disk,user can set it.

#17. jb | 2009-09-03 10:12:44, anonymous

Is is possible to make a "query-based snippets generation" in the current SVN 0.9.10 release ? How ? I really need this feature :-)

Currently, ->buildExcerpt() needs to make a database call to retrieve the docs and it's a huge bottleneck for me.

#18. pavel baranov | 2009-09-03 17:33:00, anonymous

"NEED REAL TIME UPDATES! My god, Andrew, you've been promising this for 2 years already. How much more should we wait?" AGREED !

#19. Ethaniel | 2009-09-05 08:36:22, anonymous

Ythan, how much should I pay to satisfy the developer? $500 per hour?
All I can do right now is humbly wait for years to come and keep reminding Andrew about the promise he gave us.

#20. ab | 2009-09-06 22:56:22, anonymous

Ethaniel, "humbly wait"? It doesn't seem like you are doing that at all.. If you want it that bad, do it yourself.

#21. Just wondering | 2009-09-07 08:08:13, anonymous

Any news about when 0.9.9 rc3/final is released?

#22. scott | 2009-09-09 07:49:32, anonymous

when support MySql5.1 ?

#23. Andrew Aksyonoff | 2011-05-03 13:36:41

Ethaniel, behave yourself.

#24. alonewolf | 2009-09-24 04:34:58, anonymous

qi dai zhong ....

#25. Jim | 2009-09-28 20:43:34, anonymous

Can string attributes be used to filter on, or is that going to be for a further future release like sorting and grouping?

#26. Doug | 2009-10-11 07:08:23, anonymous

Thanks for such a great program! If you have any interest in suggestions, a cool feature would be to be able to search for words appearing in the same sentence or paragraph. For instance S(blue dog) would search for a document that has blue and dog in the same sentence or P(sphinx string attributes) would search for a document that has sphinx, string, and attributes all in the same paragraph. I think those would probably return a little better results that a straight up proximity search.

#27. fdgfdg | 2009-10-19 08:41:43, anonymous

gfdgfdg

#28. Jeroen | 2009-11-04 23:27:41, anonymous

Why posting a teaser and then nothing for more than 5 months ?
Having some issues ?

#29. Just wondering | 2009-11-06 07:26:06, anonymous

That's why it's called teaser :-]

#30. DaveB | 2011-05-03 13:37:39

+1 for string attributes please! Trying to work around ordering/sorting issues is making my head hurt...

#31. Murri | 2009-12-02 17:36:50, anonymous

+1 for news on projects progress :-]

#32. Andrew | 2010-06-09 17:28:56, anonymous

Longest teaser ever (one year soon) :-)

#33. Vincent Theeten | 2011-05-03 13:37:31

any news on this yet? For the moment we've been solving this problem by BASE-N encoding our text attributes, which is ugly to say the least. String attributes would solve many of our headaches. Keep up the good work!

#34. Andrew Aksyonoff | 2011-05-03 13:36:41

Vincent, string attributes are available in trunk.

#35. Marc | 2010-08-05 23:32:01, anonymous

Hello,

we are using Sphinx 0.9.9 and are searching through almost 12 million database entries. on a quad core xeon with 2 ghz each core one query only lasts 1-3 seconds what is very fast compared to more than 45 seconds for on query with mysql fulltext search.

Thanks for the great work. I will switch to 0.9.10 when all parts of it are stable.

Good Job.

#36. Jehnavi | 2011-05-03 13:38:23

I thank the strings are stored in RAM or on disk, the user can adjust.
http://www.historicaltravelguide.com/the-great-sphinx-of-giza-2.html

Name and Comment are required; other fields are optional. Email will never be displayed nor transferred nor used for anything else except getting back in touch. Register and then login to skip spambot check and filling your name every time. Sorry, no markup at all for now.