Indexing your office documents with Elastic stack and FSCrawler
A presentation at Elastic Saudi Arabia User Group by David Pilato
Indexing your office documents with Elastic and FSCrawler y o David Pilat Developer | Evangelist, Communit @dadoonet
2
3
and getting content and metadata static void extractTextAndMetadata(InputStream stream) throws Exception BodyContentHandler handler = new BodyContentHandler() Metadata metadata = new Metadata() try (stream) new DefaultParser().parse(stream, handler, metadata, new ParseContext()) String extractedText = handler.toString() String title = metadata.get(TikaCoreProperties.TITLE) String keywords = metadata.get(TikaCoreProperties.KEYWORDS) String author = metadata.get(TikaCoreProperties.CREATOR) ; { ; ; ; ; ; ; { 4 } } Parsing a stream
n
ingest-attachment plugi extracting from BASE64 or CBOR
An ingest pipeline 6
ingest-attachment processor plugin using Tika behind the scene 7
Demo 8 https://cloud.elastic.co
r FSCrawle You know, for files…
10
Disclaime This project is a community project. It is not officially supported by Elastic. Support is only provided by FSCrawler community on discuss and stackoverflow. http://discuss.elastic.co/ https://stackoverflow.com/questions/tagged/fscrawler r 11
FSCrawle Architecture FSCrawler Local Dir JSON (noop) ES6 Mount Point XML ES7 SSH / SCP Apache Tika HTTP Rest Inputs r 12 Filters Outputs
FSCrawle Key Features • • • Much more formats than ingest attachment plugi OCR (Tesseract Much more metadata than ingest attachment plugin (See https://fscrawler.readthedocs.io/en/latest/admin/fs/elasticsearch.html#generated-fields) • Language detection n r ) 13
Documentation • • • • https://fscrawler.readthedocs.io/ https://fscrawler.readthedocs.io/en/latest/user/tutorial.html https://fscrawler.readthedocs.io/en/latest/user/formats.html https://fscrawler.readthedocs.io/en/latest/admin/fs/index.html https://fscrawler.readthedocs.io/en/latest/
Demo 15 https://cloud.elastic.co
r FSCrawle even better with a UI
FSCrawle Workplace Search integration r 17
FSCrawle Architecture FSCrawler Local Dir JSON (noop) ES6 Mount Point XML ES7 SSH / SCP Apache Tika WP7 Filters Outputs HTTP Rest Inputs r 18
19
Demo 20 https://cloud.elastic.co
Thanks PR are warmly welcomed! https://github.com/dadoonet/fscrawler ! 21
You have plenty of Open Office, Microsoft Office, PDF, images… documents and you may want to be able to search for their metadata and content. How can you do that?
In this talk, David will explain how Apache Tika can be used for that and how to combine this fantastic library with Elastic Stack:
-
Elasticsearch ingest-attachment plugin
-
Workplace Search connector for FSCrawler to have a ready to use and powerful user interface for your documents.
Video
Resources
The following resources were mentioned during the presentation or are useful additional information.
Buzz and feedback
Here’s what was said about this presentation on social media.
