Try out Scala. It's a superior substitute for Java in every possible way.
And even if my last statement is hyperbole, you can still use Java in your scala project for the critical parts where scala doesn't do a good enough job. Sbt, the standard scala build tool typically builds you src/main/scala and src/main/java folders. Write scala in the scala folders, java in the java folders, they can easily refer to each other, it's all seamless.
That is not true, you can program such an unmaintainable mess in Scala that is just not possible in Java.
You don't have to implement every programming feature just because it exists.
That makes Scala inferior in reality.
In my opinion the reason why Scala will never be mainstream. There has to be a balance between features and simplicity.
For a language that will "never be mainstream" it seems to be picking up steam pretty fast. We're very happy with it. Sensibly written Scala code is much easier to read than Java, and that's coming from someone who has been using Java for a long long time.
It's funny my boss was thinking of trying Scala, but the more he learned about it the more he became disillusioned.
I remember reading somewhere that although you might decide to limit use of Scala language facilities (as there are too many of them), you'll still have to deal with them when trying to understand how libraries work (for debugging purposes for example).
It's not only core libraries, all the available libraries that your project depends upon will one day or the other have to be debugged. I mean I had to debug Hibernate and it was a pain, I can't imagine debugging something written with advanced Scala features that I'm not very confident with.
Only problem is the scala community is teeny tiny, tool support isn't there (There isn't an IDE with a scala plugin that fully supports scala), pure scala libraries tend to be unreadable and lack any solid documentation (looking at you scalaz, lift, and dispatch), and I think most importantly the scala compiler is SLOW.
I've written a bunch of production Scala code, and I'm glad to have gotten away from it. Obviously to each their own.
I find scalaz quite readable, though admittedly I'm using Scala as an overcomplicated Haskell on the JVM rather than a superior Java. So my use pattern is mainly "which scalaz feature copies the haskell feature I want?" rather than "how can I use this lib to help me?" Haven't used lift or dispatch, so I'll take your word for it.
As for a small community, it hasn't bothered me. I get to cheat and use all the java libs.
Regarding compiler speed, I get the impression it's much worse on a mac than on linux (I use the latter, my colleagues use the former). On linux I've had little problem. I also tend to use `~ compile` in SBT. This will re-compile any time a file is modified, and notify you when an error occurs, so typically my code has already compiled by the time I actually want to run it. That's a really nice feature I wish I had in Haskell.
The more Scala code I write, the more readable the Scala code of others becomes. I don't think that Scala is intrinsically any less grokkable than historically-mainstream languages, it's simply that most devs have not been exposed to FP in any significant way (that describes me) and it takes some time to get your head around it.
It always strikes me that the IDE/tools complaints have a high degree of FUD-spread to them. I've had good luck with both Intellij+Scala plugin and with the Eclipse plugin. And yes, my code is in production and is substantially more complicated than "hello world".
I -love- SBT and I detest the XML-hell at the heart of Maven projects. As you say, to each their own.
Scala is a gateway drug to FP, which, given my newly-acquired prejudices is a good thing. But that said, you still have the vast universe of Java libraries at your disposal, and if you really want to, you can use Scala in an imperative fashion and still get a Java-with-fewer-warts experience. I've not been following the Java 8 roadmap very closely, but with each new release the language seems to get more complicated and uglier, with patch upon patch, a list of which, if stacked end-to-end, would stretch from Chicago to New York and would fill three Library of Congresses and many other Standard Metaphors. It's been a joy to work with a language without all that legacy rolled into it.
You're right about the compiler, though for me it's not a huge issue. My builds take a minute or two, not tens of minutes. You're also right about the docs being weak at this time. There are definitely trade-offs either way in the Java vs Scala decision. For my purposes and my skillset, Scala is a sweet spot with Python a close second.
Intellij Idea + Maven seemed pretty good, last time I had to write production Scala. I couldn't get it working with the fast background compiler daemon, which would have really helped.
I mainly used the Intellij scala plugin... It's code completion was incomplete and it identified syntax and type errors that weren't errors. This was on the latest intellij with the latest plugin.
And even if my last statement is hyperbole, you can still use Java in your scala project for the critical parts where scala doesn't do a good enough job. Sbt, the standard scala build tool typically builds you src/main/scala and src/main/java folders. Write scala in the scala folders, java in the java folders, they can easily refer to each other, it's all seamless.