At Lucid Imagination, we are changing that.

But advanced search and analytics can also be examined in terms of efficiency: how many manual steps and how much times can these tools eliminate in the retrieval process.

Yahoo’s Eric Baldeschwieler provided some great data on this topic at Apache Lucene Eurocon last October and of course we expect to hear more data like this year’s Lucene Revolution taking place May 7 through 10 at the Royal Sonesta Hotel in Boston.

Yahoo was one of the first large companies to test Hadoop. Hadoop now handles 5 million plus jobs a month on Yahoo – and juggles 170 petabytes of storage. The automated, accurate personalization enabled by the technology has been instrumental in doubling consumer engagement on Yahoo’s homepage. News clicks from the home page went up 160 percent because of better targeting. Recommended links steered through better analytics get 79 percent more attention than randomly generated links.

The company further claims that it diverts a far larger volume of spam targeted to its 450 million users than Google or Microsoft.

There is more on his presentation from October here.

Efficiency invariably will be a hot topic at this year’s event. Glenn Engstrand’s talk on analyzing social media via Big Data is one of the many that seem destined to touch on the topic.

We look forward to seeing you then.

Answering that question will increasingly emerge as of the most important issues in search, analytics and Big Data over the next few years.  Explaining what we all do for a living, in fact, will likely be one of the primary topics of hallway conversation at the Lucene Revolution developer’s conference taking place from May 7 to 10 at Boston’s Royal Sonesta Hotel (where every patron is treated like a Sonesta.)

Why? Comprehension is one of the great drivers in our industry. If investors can understand the power behind a particular technological breakthrough, more money flows to the category. A surge in money makes attracting new recruits easier. New recruits lead to breakthrough products, which in turn leads to a tidal wave of revenue and a virtuous circle.

Think back to the early 90s when the Internet was just emerging. The first wave of articles in the New York Times and other publications meticulously described then-unfamiliar concepts like application servers, spiders and crawlers. The public quickly began to grasp the power of the concept: with a computer they could be able to get all sorts of information and shop without leaving the house. And just as quickly, the language became streamlined. Discussions about push technology and words like crawler vanished.

Right now, there’s a good chance your friends, or at least one of your siblings, has heard of Big Data. But there is also a very good chance that many of them don’t know what it means or that the mental image one of them has of Big Data differs substantially from the way someone else comprehends it. Does Big Data mean corporations are collecting more data than they can handle or does it refer to the growing proliferation of online storage services like Box? Is Hadoop a line of children’s clothing and if not why the cute circus animal logo?

The world loves search and has transformed Google from a lab project to a worldwide colossus in 15 years. But the 2 billion humans on the web also understand the limits of search. Demand seems quite high for better navigational tools. The Lucid and Solr community are developing those tools, but we’re also going to have to start explaining the power of them. We’re also going to have to come up with a word set of words—search analytics, search analysis, enterprise search, intelligent search—that short circuits the conversation.

Let’s talk about it this week.

But in some ways it has also made it more difficult to find a precise answer. A simple search on NaCl on commercial search engines brings up links for the North American Cultural Laboratory, North American Coating Labs and ads for food thickeners.

We are very proud to announce that C. Lee Giles, the David Reese professor of the College of Information Sciences at Pennsylvania State University, will speak at Lucene Revolution next week on SeerSuite, an open source architecture built on Lucene/Solr for building search tools for specialized areas of research. The SeerSuite researchers have built CiteSeerX, a search engine for computer scientists; ChemXSeer, for chemistry; AckSeer for acknowledgements; and ArchSeer, a search engine for archaeology.

The idea is to leverage common functionality from Lucene/Solr while customizing each search engine so that it is optimized for the indexing and citation conventions of a particular discipline. Ideally, researchers will be able to find what they need quicker, skip junk results, and begin to see patterns that may not have emerged easily in a flood of results.

The balance of similarities and differences in the search engines being developed in SeerSuite, we think, also highlight one of the primary reasons why Lucene/Solr and Lucid Imagination are gaining ground. Everyone and every profession wants to take advantage of search, but they want to take advantage of it in their own particular way.

Lucene Revolution takes place from May 7 to 10 at the Royal Sonesta Hotel in Boston. We hope to see you then.

But recommendation engines are also essential for navigating modern web sites, particularly sprawling e-commerce sites or web properties with active comment boards. Even conventional corporate web sites can benefit from tools that will hone down the preferences of its customers. Visit a site of a conglomerate like Eaton. You can get information on locking differentials, plastic extrusion, the Mars Rover, oil exploration and zippers. But how one product might relate to another remains something of a mystery.

With a few tweaks, developers can deploy Solr and/or Lucene for building a great recommendation engine. Trey Grainger of CareerBuilder, which is both a gigantic e-commerce site and a web property with an active comment community, will discuss techniques for filtering, concept-based recommendations and creating hierarchies at 10:45 a.m. on May 10 during Lucene Revolution, the premier open source search developer conference taking place at the Royal Sonesta in Boston staring May 7. Grainger will provide tips and tricks for streamlining the customer experience at your site in a way that won’t become a laughing matter. We hope to see you then.

We recommend you register here.

Email a description of your problem to stump@lucenerevolution.com and see if you can Stump The Chump. Prizes will be awarded at the discretion of our panel, based on their combined (subjective) opinions about which questions they think stumped me the most:

• 1st Prize: $100 Amazon gift certificate
• 2nd Prize: $50 Amazon gift certificate
• 3rd Prize: $25 Amazon gift certificate

Even if you’ve already solved an interesting problem yourself: submit it to see if I can come up with a solution on the spot. Even if you can’t make it to the conference: submit your question anyway and you can watch me squirm later when we post video of the session on online.

Yes, from certain perspectives, that sounds ominous. But step back for a moment and think about the data and computing needs of the federal and state agencies that aren’t the IRS.

The Department of Agriculture is one the world’s primary repositories on data regarding crops, food production and water reserves. The Energy Information Administration compiles statistics on solar power production and tries to estimate the commercial feasibility of technologies such as algae fuel or deep-well geothermal. The USGS studies data on past earthquakes in a quest to develop programs that will be able to predict future rollers with some specificity.

The FBI, meanwhile, helped pioneer the use of digital data in law enforcement by championing fingerprints, forensics and DNA evidence. These are a few of the reasons the White House announced a plan earlier this year to provide $200 million in funding for Big Data. (That Big Data will create a substantial number of jobs in the next decade is another driver.)

At Lucene Revolution, our developers’ conference taking place from May 7 through 10 at the Royal Sonesta in Boston, you’ll have a chance to hear from some of the leading figures in the industry actively working with government agencies on the next generation of applications.

Wes Caldwell, chief architect at Intelligent Software Solutions (ISS), will deliver a talk entitled “Big Data Analytics: The New Strategic Weapon in Battling Counter-Terrorism” on Wednesday, May 9. ISS needs little introduction: it has been one of the leading vendors in analytics and federal projects for years. You can also hear from Seshu Simhadri, CTO of Global Computer Enterprises, on the GCE project.

Yes, the federal government is one of the largest generators of Big Data.  Now with the help of Lucene/Solr, it is able to put this incredibly large mass of unstructured data to work for the good of all its people.

2002 marked the first year that the amount of information stored on hard drives, CD-ROMs, flash cards and other digital devices surpassed analog technologies like paper, stone tables and record albums.  This is according to an article by Martin Hilbert and Priscila Lopez in Science.

The transformation outlined by the two authors and the rapid growth of digital data is nothing short of remarkable. In 1986, LPs accounted for 14 percent of the world’s data while audiocassettes held 12 percent and 8 percent, respectively. Digital was only an asterisk. In 1993, only 3 percent of the world’s information was stored on digital devices like hard drives or optical disks.

By 2000, at the height of the dot.com boom, digital storage had finally become a force in data storage, but it still only accounted for 25 percent of the world’s total capacity for storing information.

Now, digital data – particularly “dark” or unstructured data that can’t be categorized by conventional databases – dominates the world. The total amount of digital data in the world will come to 2.7 zettabytes in 2012, according to IDC, a 48 percent increase from 2011. 90 percent of it will be unstructured or “dark” data.  National Geographic recently reported that Americans snapped 80 billion digital photos in 2011.

So how will you find that photograph of the whole family you took three years ago? And how will your kids find it fifteen years from now?  How will an e-commerce company analyze all of their customer web shopping behavior (clicks) fast enough to market effectively to their latest whim?

Will we ever be able to get a handle on our data with this kind of spiraling growth?

Those are the kinds of questions we in the information retrieval business face every day and they are the kinds of questions that we will be discussing at Lucene Revolution, or developers’ conference taking place at the Royal Sonesta Hotel in Boston from May 7 to 10 (which, in 1993 would have been the home of the Lotus Notes Developer Conference.)

Speakers include Jermy Bently, CEO of Smartlogic, who will discuss analyzing metadata while Seshu Simhadri, CTO of Global Computer Enterprises, will expound on the GCE Big Data Cloud. Uwe Schindler of SD DataSolutions will provide information that can tell you whether your search engines are top-notch, or laggards.

Lucerne Imagination’s quick answer to these questions is, of course, yes. Tools like Lucerne and Solar effectively create a bridge between the digital world, dominated by computers, and the handwritten notes, audio clips, video images and photographs and other forms of digital data that derive from the real world.

These tools, more importantly, will further allow us to make predictions about the weather or traffic by unearthing patterns and trends in seemingly disparate pieces of information. Search will graduate from providing information to providing clues to the future.

Keeping up won’t be easy, but it certainly will be interesting.

• Moderator: Erik Hatcher, Lucid Imagination – Lucene/Solr Committer and co-author of Lucene in Action
• Panel Members:
  • Erick Erickson, Lucid Imagination – Lucene/Solr Committer
  • Eric Pugh, OpenSource Connections – co-author of Apache Solr 3 Enterprise Search Server
  • Grant Ingersoll, Lucid Imagination – Lucene/Solr Committer and co-author of Taming Text

The panel members will be doing their best to make me look like a chump while I try to answer questions submitted to stump[at]lucenerevolution.com by folks just like you.

Well, guess again. The pre-conference training sessions at Lucene Revolution, taking place at the Royal Sonesta in Boston from May 7 through May 10, are filling up fast. A session titled “Scaling Search with Big Data and Solr” has already filled and we’re now filling up a waiting list.

Don’t wait. If you’re already coming to Lucene Revolution, you’re well aware that corporate search, unstructured search and Big Data are poised to become three of the most prominent enterprise computing markets for the next ten to 15 years. Corporations are sitting on vast amounts of data and most of it—handwritten customer comments, texts, images, video streams—was not created with structured databases or retrieval in mind. The programmers and companies that figure out the most efficient and clever ways to access and understand that data could become the titan of computing.

Training courses will cover troubleshooting Solr, performance optimizations, building applications with Solr and security.

The classes cost $1,600 while a class and conference package can be obtained for the affordable rate of $2,100.

We hope to see you soon at one of the technology industry’s preeminent conferences, where the smartest and most innovative in IT come to learn the latest advancements in open source search.

Librarians catalog books by author, type and other factors and books typically come with indexes and tables of contents to help readers narrow their search. But what if you’re looking for a particular passage but can’t remember the book, or where it occurs in the book? Or which Roman Emperor—Elgabalus or Honorius—was obsessed with feeding chickens, according to The Decline and Fall of the Roman Empire? If you only have scraps of information and human memory at your disposal, the search can take hours.

Engy Ali, the project manager from the Bibliotheca Alexandria, will present a talk titled “How to Access Your Library Book Connections Using Solr” on day two of Lucene Revolution, our developers’ conference taking place from May 7 to May 10 at the Royal Sonesta in Boston. No more flipping through pages. Ali can also give an update on the Bibliotheca, the modern reincarnation of the famed library that was sacked five times between 48 B.C. and 642 A.D.

Finding your way through an analog world: it will be one of the dominant themes at the conference. Lucene/Solr and Lucid Imagination, of course, were created to help bring order and intelligence to unstructured data. Some of the other speaker and topic highlights include:

• A paper on searching and analyzing television news from Kai Chan, the instructional Technology and Database Developer with the Social Sciences Computing lab at UCLA.
• Gianugo Rabellino from Microsoft will deliver a keynote on Lucene/Solr on Windows and provide updates on working with Azure.
• A discussion on managing Japanese linguistics in Lucene and Solar by Chistian Moen, founder and CEO of Atilka.
• Seshu Simhadri, CTO, Global Computer Enterprises, will discuss efforts at the Federal agencies like the U.S. Coast Guard and the Department of Labor to employ Big Data and analytics via GCE.
• Andrew Urquhart, a principal systems engineer at Raytheon, will discuss techniques for employing Solar in mapping in his talk entitled “Search with Polygons: Another Approach to Solr Geospatial Search.”

Other industry thought leaders include Steve Kearns from Basis Technology; Eric Pugh, Principle, OpenSource Connections; David Hamson & Mou Nandi from NetDocuments; Petr Pleshachkov from EMC; Paddy Mullen, Independent Contractor; Glenn Engstrand, Sr. Software Engineer, Zoosk, Inc.; Scott Stults, Solutions Architect, Open Source; and Jermy Bently, CEO, Smartlogic.

Lucid Imagination speakers include Paul Doscher, CEO; Grant Ingersoll, Chief Scientist; Robert Muir; Erick Erickson; Mark Miller and Andrzej Bialeki.  And oh by the way at least 8 of the 30 Apache Solr/Lucene committers will be participating.

Where else can an individual make contact with so many brilliant minds in a single place at a single time?