Elasticsearch sur Azure : Make sense of your (BIG) data !
•
0 likes•3,304 views
The document is a presentation about using Elasticsearch on Azure. It introduces Elasticsearch and its features like scalability, plug and play functionality, and REST/JSON interface. It demonstrates how to deploy Elasticsearch on Azure by using unicast discovery across virtual machines or by using an Azure cloud plugin. It also shows how to scale out Elasticsearch on Azure by starting additional nodes and discusses using Elasticsearch to analyze big data on Azure.
Report
Share
Elasticsearch sur Azure : Make sense of your (BIG) data !
- 2. Elasticsearch sur Azure Make sense of your (BIG) data ! David Pilato / Thomas Conté! Développeurs - Evangélistes Elasticsearch / Microsoft @dadoonet / @tomconte StartUp
- 3. Donnez votre avis ! Depuis votre smartphone sur : http://notes.mstechdays.fr ! De nombreux lots à gagner toute les heures !!! Claviers, souris et jeux Microsoft… ! Merci de nous aider à améliorer les Techdays ! #mstechdays #elasticsearch StartUp
- 5. Windows Azure Plateforme Ouverte Back-ends Android, iOS & Node.js via Azure Mobile Services Devices Languages SDKs pour Java, Python, PHP, Ruby, Node.js WebSites pour PHP, Node.js, Python & App Gallery Apps MySQL via ClearDB, MongoDB via MongoLab, Hadoop via HDInsight Operating Systems Linux VMs via Image Gallery & VMDepot #mstechdays #elasticsearch Data StartUp
- 6. Windows Azure : rappels Identity Cloud Services Table Storage Blob Storage 3rd Party Services Big Data Integration “Plateforme Ouverte” Virtual Machines Web Sites Caching Service Bus #mstechdays #elasticsearch SQL Azure Analytics Mobile Services Media Services StartUp CDN
- 8. search = like % ? #mstechdays #elasticsearch StartUp
- 9. search = like % ? SELECT doc.*, pays.* FROM doc, pays WHERE doc.pays_code = pays.code AND doc.date_doc > to_date('2011-12', 'yyyy-mm') AND doc.date_doc < to_date('2012-01', 'yyyy-mm') AND lower(pays.libelle) = 'france' AND lower(doc.commentaire) LIKE ‘%produit%' AND lower(doc.commentaire) LIKE ‘%david%'; #mstechdays #elasticsearch StartUp
- 10. Moteur de recherche ? #mstechdays #elasticsearch StartUp
- 11. Moteur de recherche ? #mstechdays #elasticsearch StartUp
- 16. plug & play scalable elasticsearch ? REST/JSON #mstechdays #elasticsearch Lucene StartUp
- 17. plug & play scalable elasticsearch Apache 2 license REST/JSON #mstechdays #elasticsearch Lucene StartUp
- 18. Start… $ wget https://download.elasticsearch.org/elasticsearch/ elasticsearch/elasticsearch-1.0.0.tar.gz! $ tar -xf elasticsearch-1.0.0.tar.gz! $ ./elasticsearch-1.0.0/bin/elasticsearch! [INFO ][node ][Ghost Maker] {1.0.0}[5645]: initializing #mstechdays #elasticsearch StartUp
- 19. … and play! $ curl -XPUT localhost:9200/sessions/session/1 -d '{! "title" : "Elasticsearch sur Azure",! "subtitle" : "Make sense of your (BIG) data !",! "date" : "2014-02-13T16:30:00",! "tags" : [ "elasticsearch", "azure", "cloud" ],! "speaker" : [ {! "first_name" : "Thomas", ! "last_name" : "Conté" ! }, {! "first_name" : "David", ! "last_name" : "Pilato" ! }]! }' #mstechdays #elasticsearch StartUp
- 20. Search! $ curl -XPOST http://localhost:9200/sessions/session/_search -d' { "query": { "multi_match": { "query": "elasticsearch azure thomas", "fields": [ "title^2", "speaker.first_name" ] } }, "post_filter": { "range": { "date": { "from": "2014-02-13", "to": "2014-02-14" } } } }' #mstechdays #elasticsearch StartUp
- 22. Compute! $ curl -XPOST http://localhost:9200/sessions/session/_search -d' { "query": { ... }, "aggs": { "by_date": { "date_histogram": { "field": "date", "interval": "day", "format" : "dd/MM/yyyy" } } } }' #mstechdays #elasticsearch StartUp
- 23. Compute! $ curl -XPOST http://localhost:9200/sessions/session/_search -d' { "query": { ... }, "aggs": { "by_date": { "date_histogram": { "field": "date", "interval": "day", "format" : "dd/MM/yyyy" } } "by_date": [ } { "key_as_string": "11/02/2014", "doc_count": 1 }, }' { "key_as_string": "12/02/2014", "doc_count": 2 }, { "key_as_string": "13/02/2014", "doc_count": 3 } ] #mstechdays #elasticsearch StartUp
- 24. Compute! $ curl -XPOST http://localhost:9200/sessions/session/_search -d' { "query": { ... }, "aggs": { "by_date": { "date_histogram": { "field": "date", "interval": "day", "format" : "dd/MM/yyyy" } } "by_date": [ } { "key_as_string": "11/02/2014", "doc_count": 1 }, }' { "key_as_string": "12/02/2014", "doc_count": 2 }, { "key_as_string": "13/02/2014", "doc_count": 3 } ] #mstechdays #elasticsearch StartUp
- 25. démo MAKE SENSE OF YOUR (BIG) DATA! let’s inject some marketing documents… #mstechdays #elasticsearch StartUp
- 26. ELASTICSEARCH Elastique ? Distribué ? #mstechdays #elasticsearch StartUp
- 27. Distributed indices node 1 #mstechdays #elasticsearch StartUp
- 28. Distributed indices node 1 $ curl -XPUT localhost:9200/orders -d '{! "settings.index.number_of_shards" : 4,! "settings.index.number_of_replicas" : 1! }' #mstechdays #elasticsearch StartUp
- 29. Distributed indices node 1 orders $ curl -XPUT localhost:9200/orders -d '{! "settings.index.number_of_shards" : 4,! "settings.index.number_of_replicas" : 1! }' #mstechdays #elasticsearch StartUp
- 30. Distributed indices node 1 orders 1 2 3 4 #mstechdays #elasticsearch $ curl -XPUT localhost:9200/orders -d '{! "settings.index.number_of_shards" : 4,! "settings.index.number_of_replicas" : 1! }' StartUp
- 31. Distributed indices node 1 orders 1 2 3 4 $ curl -XPUT localhost:9200/orders -d '{! "settings.index.number_of_shards" : 4,! "settings.index.number_of_replicas" : 1! }' $ curl -XPUT localhost:9200/products -d '{! "settings.index.number_of_shards" : 2,! "settings.index.number_of_replicas" : 0! }' #mstechdays #elasticsearch StartUp
- 32. Distributed indices node 1 orders 1 2 3 4 products 1 2 #mstechdays #elasticsearch $ curl -XPUT localhost:9200/orders -d '{! "settings.index.number_of_shards" : 4,! "settings.index.number_of_replicas" : 1! }' $ curl -XPUT localhost:9200/products -d '{! "settings.index.number_of_shards" : 2,! "settings.index.number_of_replicas" : 0! }' StartUp
- 33. Start new nodes $ ./elasticsearch-1.0.0/bin/elasticsearch! [INFO ][node ][Armageddon] {1.0.0}[5645]: initializing [INFO ][cluster.service][Armageddon] detected_master [Ghost Maker] #mstechdays #elasticsearch StartUp
- 34. Start new nodes $ ./elasticsearch-1.0.0/bin/elasticsearch! [INFO ][node ][Armageddon] {1.0.0}[5645]: initializing [INFO ][cluster.service][Armageddon] detected_master [Ghost Maker] $ ./elasticsearch-1.0.0/bin/elasticsearch! [INFO ][node ][Cyclops] {1.0.0}[5645]: initializing [INFO ][cluster.service][Cyclops] detected_master [Ghost Maker] #mstechdays #elasticsearch StartUp
- 35. Start new nodes $ ./elasticsearch-1.0.0/bin/elasticsearch! [INFO ][node ][Armageddon] {1.0.0}[5645]: initializing [INFO ][cluster.service][Armageddon] detected_master [Ghost Maker] $ ./elasticsearch-1.0.0/bin/elasticsearch! [INFO ][node ][Cyclops] {1.0.0}[5645]: initializing [INFO ][cluster.service][Cyclops] detected_master [Ghost Maker] $ ./elasticsearch-1.0.0/bin/elasticsearch! [INFO ][node ][Karnak] {1.0.0}[5645]: initializing [INFO ][cluster.service][Karnak] detected_master [Ghost Maker] #mstechdays #elasticsearch StartUp
- 36. Distributed indices node 1 orders 1 2 3 4 products 1 2 #mstechdays #elasticsearch StartUp
- 37. Distributed indices node 1 orders 1 2 3 4 products 1 2 #mstechdays #elasticsearch $ ./elasticsearch-1.0.0/bin/elasticsearch! [INFO ][node ][Armageddon] {1.0.0}[5645]: initializing [INFO ][cluster.service][Armageddon] detected_master [Ghost Maker] StartUp
- 38. Distributed indices node 1 node 2 orders 1 2 3 4 products 1 2 #mstechdays #elasticsearch $ ./elasticsearch-1.0.0/bin/elasticsearch! [INFO ][node ][Armageddon] {1.0.0}[5645]: initializing [INFO ][cluster.service][Armageddon] detected_master [Ghost Maker] StartUp
- 39. Distributed indices node 1 node 2 orders orders 1 2 3 4 products 1 products 2 #mstechdays #elasticsearch StartUp
- 40. Distributed indices node 1 node 2 orders orders 1 2 1 2 3 4 3 4 products 1 products 2 #mstechdays #elasticsearch StartUp
- 41. Distributed indices node 1 node 2 orders orders 1 2 1 2 3 4 3 4 products 1 #mstechdays #elasticsearch products 2 StartUp
- 42. Distributed indices node 1 node 2 orders orders 1 2 1 2 3 4 3 4 products 1 #mstechdays #elasticsearch products $ ./elasticsearch-1.0.0/bin/elasticsearch! 2 [INFO ][node ][Karnak] {1.0.0}[5645]: initializing [INFO ][cluster.service][Karnak] detected_master [Ghost Maker] StartUp
- 43. Distributed indices node 1 node 2 orders node 3 orders 1 2 1 2 3 4 3 4 products 1 #mstechdays #elasticsearch products $ ./elasticsearch-1.0.0/bin/elasticsearch! 2 [INFO ][node ][Karnak] {1.0.0}[5645]: initializing [INFO ][cluster.service][Karnak] detected_master [Ghost Maker] StartUp
- 44. Distributed indices node 1 node 2 orders node 3 orders orders 1 2 1 2 3 4 3 4 products 1 #mstechdays #elasticsearch products products 2 StartUp
- 45. Distributed indices node 1 node 2 orders 1 node 3 orders 2 4 products 1 #mstechdays #elasticsearch orders 2 1 3 3 products products 2 StartUp 4
- 46. ELASTICSEARCH SUR AZURE Discovery ? #mstechdays #elasticsearch StartUp
- 47. Deploying on Azure (unicast) Cloud service: your-es.cloudapp.net VM node 1 private IP1 VM node 2 private IP2 discovery.zen.ping.multicast.enabled: false! discovery.zen.ping.unicast.hosts: ["ip1", "ip2"] #mstechdays #elasticsearch StartUp
- 48. Deploying on Azure (unicast) Cloud service: your-es.cloudapp.net VM node 1 private IP1 VM node 2 private IP2 VM node 3 private IP3 discovery.zen.ping.multicast.enabled: false! discovery.zen.ping.unicast.hosts: ["ip1", "ip2", "ip3"] #mstechdays #elasticsearch StartUp
- 49. Deploying on Azure (cloud plugin) Cloud service: your-es.cloudapp.net VM node 1 private IP1 VM node 2 private IP2 Azure REST API #mstechdays #elasticsearch StartUp VM node 3 private IP3
- 50. Deploying on Azure (cloud plugin) Cloud service: your-es.cloudapp.net VM node 1 private IP1 Azure REST API VM node 2 private IP2 cloud:! azure:! keystore: /path/to/keystore! password: your_password_for_keystore! subscription_id: your_azure_subscription_id! service_name: your-es! discovery:! type: azure #mstechdays #elasticsearch StartUp VM node 3 private IP3
- 52. démo ELASTICSEARCH ON AZURE Scale out! #mstechdays #elasticsearch StartUp
- 53. Digital is business QUESTIONS ? @dadoonet / @tomconte © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
- 54. Digital is business QUESTIONS ? @dadoonet / @tomconte © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.