-
ELK스택 기본사용법 - ① ElasticSearch데이터베이스 2022. 10. 26. 13:46
ELK (Elastic search, Logstash, Kibana) 스택에 대해서 들어보기는 했지만 어떻게 동작하는지 몰라서 간단한 강의를 듣고 실습해 보았다!
학습한 내용과 사진, 코드는 모두 다음의 강의에서 참고하였다!🙌
참고한 강의 (플레이리스트)
↓
https://www.youtube.com/watch?v=3iA-ncqAqYE&list=PLVNY1HnUlO24LCsgOxR_eK2Yi4sOgH9Pg&index=19
github : 강의 참고 자료
↓
https://github.com/minsuk-heo/BigData
GitHub - minsuk-heo/BigData
Contribute to minsuk-heo/BigData development by creating an account on GitHub.
github.com
ELK란?
빅데이터를 다루는 툴로서 분석 및 저장을 담당하는 ElasticSearch, 수집을 담당하는 Logstash, 수집된 데이터를 시각화하는 Kibana로 구성된다. 즉, 데이터 수집 - 분석 툴로서 높은 확장성과 용이성을 지닌다. 추가로 ElasticSearch의 경우 빠른 검색도 가능하다.
ElasticSearch
Lucene 기반으로 개발된 분산 검색 엔진으로, Logstash를 통해 수집된 데이터를 저장소에 저장하는 역할을 담당한다. RDB와 비슷하지만 검색 속도 및 분석 성능이 훨씬 뛰어나다. 텍스트 검색시 관계형 데이터의 시간 복잡도는 O(n)인 반면 엘라스틱 서치는 해시 테이블과 같이 O(1)의 시간이 걸린다.
설치하기
먼저 ElasticSearch를 사용하기 위해 설치를 진행한다. macOS를 사용하고 있어 brew 명령어를 이용하여 설치를 진행했다.
참고사이트
↓
https://memostack.tistory.com/201
MacOSX 에 ElasticSearch 설치하기 (Homebrew 사용)
Elastic Search 설치 아래 사이트에서 설치를 진행한다. www.elastic.co/kr/downloads/elasticsearch Download Elasticsearch Free | Get Started Now | Elastic | Elastic Want it hosted? Deploy on Elastic Clo..
memostack.tistory.com
$ brew tap elastic/tap $ brew install elastic/tap/elasticsearch-full다음 명령어로 ElasticSearch를 실행할 수 있다.
$ brew services start elastic/tap/elasticsearch-fullcurl 명령어를 이용해 올바르게 서비스가 실행중인지 확인한다.
$ curl http://localhost:9200만약 여기서 Failed to connect to localhost port 9200 after 4 ms: Connection refused 문구가 발생한다면
curl http://127.0.0.1:9200으로 시도하면 서비스 실행을 확인할 수 있다.
% curl http://127.0.0.1:9200 { "name" : ["컴퓨터 이름"], "cluster_name" : ["생성된 클러스터이름"], "cluster_uuid" : ["클러스터 UUID"], "version" : { "number" : "7.17.4", "build_flavor" : "default", "build_type" : "tar", "build_hash" : "79878662c54c886ae89206c685d9f1051a9d6411", "build_date" : "2022-05-18T18:04:20.964345128Z", "build_snapshot" : false, "lucene_version" : "8.11.1", "minimum_wire_compatibility_version" : "6.8.0", "minimum_index_compatibility_version" : "6.0.0-beta1" }, "tagline" : "You Know, for Search" }서비스 종료는 다음의 명령어로 실행할 수 있다.
$ brew services stop elastic/tap/elasticsearch-full+ 추가적으로 ElasticSearch 정보의 위치를 알아보자.
home : elasticsearch의 홈 디렉토리 위치
/usr/local/var/homebrew/linked/elasticsearch-fullbin : elasticsearch의 바이너리 위치
/usr/local/var/homebrew/linked/elasticsearch-full/binconf : 설정 파일 위치 (elasticsearch.yml 파일 등)
/usr/local/etc/elasticsearchdata : 데이터 파일 위치 ( 경로 변경 시 path.data를 수정)
/usr/local/var/lib/elasticsearchJAVAlogs : 로그 파일 위치 (경로 변경 시 path.logs를 수정)
/usr/local/var/log/elasticsearchJAVAplugins : 플러그인 위치
/usr/local/var/homebrew/linked/elasticsearch/pluginsJAVA데이터 다루기(CRUD -> GET, POST, PUT, DELETE)
1) 인덱스
엘라스틱 서치는 데이터를 다루는데에 HTTP를 사용한다.
- 인덱스( = 관계형 DB의 데이터베이스) 조회
curl -X GET 'http://localhost:9200/[인덱스]?pretty'먼저 인덱스가 존재하는지 확인해보자. 현재 해당 인덱스가 없기 때문에 404를 반환한다. (결과값을 더 깔끔하고 예쁘기 위해 pretty를 사용하였다!)
?pretty에서 오류가 발생한다면 '을 넣어주어야 한다!
참고한 사이트
↓
zsh: no matches found when $ curl -s -X localhost:60702/api/bundle?name=light%20reading | j q '.'
I am working through the Node.js The Right Way book by Jim Wilson. I am currently trying to use a PUSH request to create a new bundle with the specified name. * curl -X POST http://:/api/bundle?na...
stackoverflow.com
% curl -X GET 'http://localhost:9200/classes?pretty' { "error" : { "root_cause" : [ { "type" : "index_not_found_exception", "reason" : "no such index [classes]", "resource.type" : "index_or_alias", "resource.id" : "classes", "index_uuid" : "_na_", "index" : "classes" } ], "type" : "index_not_found_exception", "reason" : "no such index [classes]", "resource.type" : "index_or_alias", "resource.id" : "classes", "index_uuid" : "_na_", "index" : "classes" }, "status" : 404 }- 인덱스( = 관계형 DB의 데이터베이스) 생성
curl -XPUT http://localhost:9200/[생성하고자 하는 인덱스]인덱스를 생성하기 위해서는 PUT을 사용한다. 반환되는 'acknowledged: true'는 인덱스가 생성되었음을 나타낸다.
% curl -X PUT 'http://localhost:9200/classes' { "acknowledged":true, "shards_acknowledged":true, "index":"classes" }다시 조회하면 인덱스가 생성되는 것을 확인할 수 있다.
% curl -X GET 'http://localhost:9200/classes?pretty' { "classes" : { "aliases" : { }, "mappings" : { }, "settings" : { "index" : { "routing" : { "allocation" : { "include" : { "_tier_preference" : "data_content" } } }, "number_of_shards" : "1", "provided_name" : "classes", "creation_date" : "1666921085529", "number_of_replicas" : "1", "uuid" : "sG59Mt11RDa-xU3pfvOkyg", "version" : { "created" : "7170499" } } } } }- 인덱스( = 관계형 DB의 데이터베이스) 삭제
curl -X DELETE 'http://localhost:9200/[인덱스]'생성된 인덱스를 삭제하는 것은 다음과 같다.
% curl -X DELETE 'http://localhost:9200/classes' { "acknowledged":true }다시 삭제한 인덱스를 조회하면 인덱스가 존재하지 않기 때문에 404를 반환한다.
% curl -X GET 'http://localhost:9200/classes?pretty' { "error" : { "root_cause" : [ { "type" : "index_not_found_exception", "reason" : "no such index [classes]", "resource.type" : "index_or_alias", "resource.id" : "classes", "index_uuid" : "_na_", "index" : "classes" } ], "type" : "index_not_found_exception", "reason" : "no such index [classes]", "resource.type" : "index_or_alias", "resource.id" : "classes", "index_uuid" : "_na_", "index" : "classes" }, "status" : 404 }2) Document
- Document( = 관계형 DB의 Row) 생성
curl -XPOST http://localhost:9200/[인덱스명]/[타입명]/[아이디]/ -H 'Content-Type: application/json' -d '[데이터]'||@[데이터파일]이번엔 json 데이터를 이용하여 POST 메서드를 통해 인덱스에 document를 생성한다.
위와 같이 헤더타입을 -H 'Content-Type: application/json'로 지정하지 않고 요청을 보내면, 다음과 같은 406 에러가 발생한다.
참고한 사이트
↓
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
If you see the following error when using curl, it means it does not understand the format in the file. Here we're trying to create an index in elasticsearch using a file. curl -s -XPOST localhost:9200/products/_bulk --data-binary @products.json {"error":.
www.lightnetics.com
% curl -X POST http://localhost:9200/classes/class/1/ -d '{"title" : "Algorithm", "professor" : "John"}' { "error":"Content-Type header [application/x-www-form-urlencoded] is not supported", "status":406 }헤더타입을 올바르게 입력하고 요청하면,
% curl -X POST http://localhost:9200/classes/class/1/ -H 'Content-Type: application/json' -d '{"title" : "Algorithm", "professor" : "John"}' { "_index":"classes", "_type":"class", "_id":"1","_version":1, "result":"created", "_shards":{ "total":2, "successful":1, "failed":0 }, "_seq_no":0, "_primary_term":1 }잘 조회되는 것을 확인할 수 있다.
% curl -X GET 'http://localhost:9200/classes/class/1?pretty' { "_index" : "classes", "_type" : "class", "_id" : "1", "_version" : 1, "_seq_no" : 0, "_primary_term" : 1, "found" : true, "_source" : { "title" : "Algorithm", "professor" : "John" } }여기서 json 텍스트가 아닌 .json 타입의 파일을 통한 document 생성시에는 다음과 같이 사용할 수 있다.
먼저 다음과 같이 . json 파일을 만들어준다.
% vi oneclass.json { "title" : "Machine Learning", "Professor" : "Minsuk Heo", "major" : "Computer Science", "semester" : ["spring", "fall"], "student_count" : 100, "unit" : 3, "rating" : 5 }다음과 같이 파일을 통해서 요청보내면,
% curl -X POST http://localhost:9200/classes/class/1/ -H 'Content-Type: application/json' -d @oneclass.json { "_index":"classes", "_type":"class", "_id":"1", "_version":2, "result":"updated", "_shards":{ "total":2, "successful":1, "failed":0 }, "_seq_no":1, "_primary_term":1 }잘 조회되는 것을 확인할 수 있다.
% curl -X GET 'http://localhost:9200/classes/class/1?pretty' { "_index" : "classes", "_type" : "class", "_id" : "1", "_version" : 2, "_seq_no" : 1, "_primary_term" : 1, "found" : true, "_source" : { "title" : "Machine Learning", "Professor" : "Minsuk Heo", "major" : "Computer Science", "semester" : [ "spring", "fall" ], "student_count" : 100, "unit" : 3, "rating" : 5 } }- Document( = 관계형 DB의 Row) 수정
curl -XPOST http://localhost:9200/[인덱스명]/[타임명]/[아이디]/_update -d '{"doc" : [데이터]}'이번엔 _update 옵션을 사용하고 doc 프로퍼티를 이용하여 기존에 생성된 document에 하나의 필드를 추가하는 업데이트를 진행해보자.
% curl -X POST 'http://localhost:9200/classes/class/1/_update?pretty' -H 'Content-Type: application/json' -d '{"doc":{"unit":1}}' { "_index" : "classes", "_type" : "class", "_id" : "1", "_version" : 3, "result" : "updated", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 2, "_primary_term" : 1 }조회시 unit 필드가 추가된 것을 확인할 수 있다.
% curl -X GET 'http://localhost:9200/classes/class/1?pretty' { "_index" : "classes", "_type" : "class", "_id" : "1", "_version" : 3, "_seq_no" : 2, "_primary_term" : 1, "found" : true, "_source" : { "title" : "Machine Learning", "Professor" : "Minsuk Heo", "major" : "Computer Science", "semester" : [ "spring", "fall" ], "student_count" : 100, "unit" : 1, "rating" : 5 } }이번엔 추가했던 필드의 값을 수정해보자. 위와 방법은 동일하다.
% curl -X POST 'http://localhost:9200/classes/class/1/_update?pretty' -H 'Content-Type: application/json' -d '{"doc":{"unit":2}}' { "_index" : "classes", "_type" : "class", "_id" : "1", "_version" : 4, "result" : "updated", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 3, "_primary_term" : 1 }다시 조회해보면 unit필드의 값이 2로 업데이트 된 것을 확인할 수 있다.
% curl -X GET 'http://localhost:9200/classes/class/1?pretty' { "_index" : "classes", "_type" : "class", "_id" : "1", "_version" : 4, "_seq_no" : 3, "_primary_term" : 1, "found" : true, "_source" : { "title" : "Machine Learning", "Professor" : "Minsuk Heo", "major" : "Computer Science", "semester" : [ "spring", "fall" ], "student_count" : 100, "unit" : 2, "rating" : 5 } }이번에 스크립트를 이용하여 기존값을 이용한 변경이 이루어지도록 해보자.
curl -X POST 'http://localhost:9200/classes/class/1/_update?pretty' -H 'Content-Type: application/json' -d '{"script":"ctx._source.[필드]+=[추가값]"}'기존 unit 필드의 값에 5를 추가하는 요청을 보낸다.
% curl -X POST 'http://localhost:9200/classes/class/1/_update?pretty' -H 'Content-Type: application/json' -d '{"script":"ctx._source.unit+=5"}' { "_index" : "classes", "_type" : "class", "_id" : "1", "_version" : 5, "result" : "updated", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 4, "_primary_term" : 1 }다시 조회해보면 unit 필드의 값이 기존 2였던 값에 5가 추가된 7로 업데이트 된 것을 확인할 수 있다.
% curl -X GET 'http://localhost:9200/classes/class/1?pretty' { "_index" : "classes", "_type" : "class", "_id" : "1", "_version" : 5, "_seq_no" : 4, "_primary_term" : 1, "found" : true, "_source" : { "title" : "Machine Learning", "Professor" : "Minsuk Heo", "major" : "Computer Science", "semester" : [ "spring", "fall" ], "student_count" : 100, "unit" : 7, "rating" : 5 } }- Document( = 관계형 DB의 Row) 벌크 생성
curl -XPOST http://localhost:9200/_bulk?pretty --data-binary @[파일documents]bulk는 여러개의 document를 한번에 생성해주는 역할을 수행한다.
먼저 다음과 같이 넣어줄 데이터를 가진 .json 파일을 만들어준다.
% vi classes.json { "index" : { "_index" : "classes", "_type" : "class", "_id" : "1" } } {"title" : "Machine Learning","Professor" : "Minsuk Heo","major" : "Computer Science","semester" : ["spring", "fall"],"student_count" : 100,"unit" : 3,"rating" : 5, "submit_date" : "2016-01-02", "school_location" : {"lat" : 36.00, "lon" : -120.00}} { "index" : { "_index" : "classes", "_type" : "class", "_id" : "2" } } {"title" : "Network","Professor" : "Minsuk Heo","major" : "Computer Science","semester" : ["fall"],"student_count" : 50,"unit" : 3,"rating" : 4, "submit_date" : "2016-02-02", "school_location" : {"lat" : 36.00, "lon" : -120.00}} { "index" : { "_index" : "classes", "_type" : "class", "_id" : "3" } } {"title" : "Operating System","Professor" : "Minsuk Heo","major" : "Computer Science","semester" : ["spring"],"student_count" : 50,"unit" : 3,"rating" : 4, "submit_date" : "2016-03-02", "school_location" : {"lat" : 36.00, "lon" : -120.00}} { "index" : { "_index" : "classes", "_type" : "class", "_id" : "5" } } {"title" : "Machine Learning","Professor" : "Tim Cook","major" : "Computer Science","semester" : ["spring"],"student_count" : 40,"unit" : 3,"rating" : 2, "submit_date" : "2016-04-02", "school_location" : {"lat" : 39.00, "lon" : -112.00}} { "index" : { "_index" : "classes", "_type" : "class", "_id" : "6" } } {"title" : "Network","Professor" : "Tim Cook","major" : "Computer Science","semester" : ["summer"],"student_count" : 30,"unit" : 3,"rating" : 2, "submit_date" : "2016-02-02", "school_location" : {"lat" : 36.00, "lon" : -120.00}} { "index" : { "_index" : "classes", "_type" : "class", "_id" : "7" } } {"title" : "Operating System","Professor" : "Jeniffer Anderson","major" : "Computer Science","semester" : ["winter"],"student_count" : 30,"unit" : 3,"rating" : 1, "submit_date" : "2016-11-02", "school_location" : {"lat" : 39.97, "lon" : -89.78}} { "index" : { "_index" : "classes", "_type" : "class", "_id" : "8" } } {"title" : "Algorithm","Professor" : "Tim Cook","major" : "Computer Science","semester" : ["fall"],"student_count" : 80,"unit" : 3,"rating" : 2, "submit_date" : "2016-10-22", "school_location" : {"lat" : 39.97, "lon" : -89.78}} { "index" : { "_index" : "classes", "_type" : "class", "_id" : "9" } } {"title" : "Data Structure","Professor" : "Tim Cook","major" : "Computer Science","semester" : ["winter"],"student_count" : 50,"unit" : 3,"rating" : 2, "submit_date" : "2016-07-22", "school_location" : {"lat" : 39.97, "lon" : -89.78}} { "index" : { "_index" : "classes", "_type" : "class", "_id" : "10" } } {"title" : "Computer Graphic","Professor" : "Jeniffer Anderson","major" : "Computer Science","semester" : ["spring"],"student_count" : 60,"unit" : 2,"rating" : 3, "submit_date" : "2016-11-12", "school_location" : {"lat" : 39.97, "lon" : -89.78}} { "index" : { "_index" : "classes", "_type" : "class", "_id" : "11" } } {"title" : "Music Fundamental","Professor" : "Jay Z","major" : "Music","semester" : ["fall"],"student_count" : 100,"unit" : 3,"rating" : 5, "submit_date" : "2016-05-22", "school_location" : {"lat" : 42.51, "lon" : -74.83}} { "index" : { "_index" : "classes", "_type" : "class", "_id" : "12" } } {"title" : "Vocal Techniques","Professor" : "Beyonce","major" : "Music","semester" : ["fall"],"student_count" : 30,"unit" : 3,"rating" : 5, "submit_date" : "2016-11-22", "school_location" : {"lat" : 42.51, "lon" : -74.83}} { "index" : { "_index" : "classes", "_type" : "class", "_id" : "13" } } {"title" : "Guitar Techiniques","Professor" : "Eric Clapton","major" : "Music","semester" : ["spring", "fall"],"student_count" : 20,"unit" : 2,"rating" : 4, "submit_date" : "2016-03-12", "school_location" : {"lat" : 42.51, "lon" : -74.83}} { "index" : { "_index" : "classes", "_type" : "class", "_id" : "14" } } {"title" : "Finance","Professor" : "Bill Gates","major" : "Accounting","semester" : ["winter"],"student_count" : 50,"unit" : 3,"rating" : 2, "submit_date" : "2016-01-12", "school_location" : {"lat" : 42.51, "lon" : -74.83}} { "index" : { "_index" : "classes", "_type" : "class", "_id" : "15" } } {"title" : "Marketing","Professor" : "Bill Gates","major" : "Accounting","semester" : ["spring"],"student_count" : 60,"unit" : 2,"rating" : 3, "submit_date" : "2016-01-22", "school_location" : {"lat" : 42.51, "lon" : -74.83}} { "index" : { "_index" : "classes", "_type" : "class", "_id" : "16" } } {"title" : "Accounting Information Systems","Professor" : "Tom Cruise","major" : "Accounting","semester" : ["fall"],"student_count" : 100,"unit" : 2,"rating" : 1, "submit_date" : "2016-11-12", "school_location" : {"lat" : 42.51, "lon" : -74.83}} { "index" : { "_index" : "classes", "_type" : "class", "_id" : "17" } } {"title" : "Individual Taxation","Professor" : "Tom Cruise","major" : "Accounting","semester" : ["fall"],"student_count" : 30,"unit" : 1,"rating" : 2, "submit_date" : "2016-08-02", "school_location" : {"lat" : 42.32, "lon" : -94.74}} { "index" : { "_index" : "classes", "_type" : "class", "_id" : "18" } } {"title" : "Auditing","Professor" : "Victoria Park","major" : "Accounting","semester" : ["spring", "fall"],"student_count" : 20,"unit" : 2,"rating" : 3, "submit_date" : "2016-09-13", "school_location" : {"lat" : 42.32, "lon" : -94.74}} { "index" : { "_index" : "classes", "_type" : "class", "_id" : "19" } } {"title" : "Cell Biology","Professor" : "Anjella Kim","major" : "Medical","semester" : ["fall"],"student_count" : 40,"unit" : 5,"rating" : 5, "submit_date" : "2016-02-22", "school_location" : {"lat" : 42.32, "lon" : -94.74}} { "index" : { "_index" : "classes", "_type" : "class", "_id" : "20" } } {"title" : "Physiology","Professor" : "Jack Berk","major" : "Medical","semester" : ["summer"],"student_count" : 30,"unit" : 5,"rating" : 4, "submit_date" : "2016-11-12", "school_location" : {"lat" : 32.69, "lon" : -99.44}} { "index" : { "_index" : "classes", "_type" : "class", "_id" : "21" } } {"title" : "Neuroscience","Professor" : "Jihee Yang","major" : "Medical","semester" : ["spring", "fall"],"student_count" : 20,"unit" : 5,"rating" : 4, "submit_date" : "2016-06-03", "school_location" : {"lat" : 32.69, "lon" : -99.44}} { "index" : { "_index" : "classes", "_type" : "class", "_id" : "22" } } {"title" : "Immunology","Professor" : "Meredith Lee","major" : "Medical","semester" : ["winter"],"student_count" : 30,"unit" : 3,"rating" : 2, "submit_date" : "2016-06-21", "school_location" : {"lat" : 32.69, "lon" : -99.44}} { "index" : { "_index" : "classes", "_type" : "class", "_id" : "23" } } {"title" : "Genetics","Professor" : "David Pollack","major" : "Medical","semester" : ["spring"],"student_count" : 20,"unit" : 3,"rating" : 3, "submit_date" : "2016-06-30", "school_location" : {"lat" : 28.22, "lon" : -81.87}} { "index" : { "_index" : "classes", "_type" : "class", "_id" : "24" } } {"title" : "Biochemistry","Professor" : "John Miller","major" : "Medical","semester" : ["fall"],"student_count" : 30,"unit" : 3,"rating" : 4, "submit_date" : "2016-01-11", "school_location" : {"lat" : 28.22, "lon" : -81.87}} { "index" : { "_index" : "classes", "_type" : "class", "_id" : "25" } } {"title" : "Anatomy","Professor" : "Tom Johnson","major" : "Medical","semester" : ["fall"],"student_count" : 30,"unit" : 5,"rating" : 3, "submit_date" : "2016-11-12", "school_location" : {"lat" : 28.22, "lon" : -81.87}}이제 _bulk 를 이용하여 정보를 한번에 주입한다.
% curl -X POST 'http://localhost:9200/_bulk?pretty' -H 'Content-Type: application/json' --data-binary @classes.json { "took" : 259, "errors" : false, "items" : [ { "index" : { "_index" : "classes", "_type" : "class", "_id" : "1", "_version" : 6, "result" : "updated", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 5, "_primary_term" : 1, "status" : 200 } }, { "index" : { "_index" : "classes", "_type" : "class", "_id" : "2", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 6, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "classes", "_type" : "class", "_id" : "3", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 7, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "classes", "_type" : "class", "_id" : "5", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 8, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "classes", "_type" : "class", "_id" : "6", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 9, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "classes", "_type" : "class", "_id" : "7", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 10, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "classes", "_type" : "class", "_id" : "8", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 11, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "classes", "_type" : "class", "_id" : "9", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 12, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "classes", "_type" : "class", "_id" : "10", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 13, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "classes", "_type" : "class", "_id" : "11", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 14, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "classes", "_type" : "class", "_id" : "12", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 15, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "classes", "_type" : "class", "_id" : "13", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 16, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "classes", "_type" : "class", "_id" : "14", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 17, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "classes", "_type" : "class", "_id" : "15", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 18, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "classes", "_type" : "class", "_id" : "16", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 19, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "classes", "_type" : "class", "_id" : "17", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 20, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "classes", "_type" : "class", "_id" : "18", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 21, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "classes", "_type" : "class", "_id" : "19", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 22, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "classes", "_type" : "class", "_id" : "20", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 23, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "classes", "_type" : "class", "_id" : "21", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 24, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "classes", "_type" : "class", "_id" : "22", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 25, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "classes", "_type" : "class", "_id" : "23", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 26, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "classes", "_type" : "class", "_id" : "24", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 27, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "classes", "_type" : "class", "_id" : "25", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 28, "_primary_term" : 1, "status" : 201 } } ] }아이디를 이용해 하나의 정보를 조회하면 알맞게 정보를 가져오는 것을 확인할 수 있다.
% curl -X GET 'http://localhost:9200/classes/class/5?pretty' { "_index" : "classes", "_type" : "class", "_id" : "5", "_version" : 1, "_seq_no" : 8, "_primary_term" : 1, "found" : true, "_source" : { "title" : "Machine Learning", "Professor" : "Tim Cook", "major" : "Computer Science", "semester" : [ "spring" ], "student_count" : 40, "unit" : 3, "rating" : 2, "submit_date" : "2016-04-02", "school_location" : { "lat" : 39.0, "lon" : -112.0 } } }Mapping
curl -XPUT 'http://localhost:9200/[인덱스]/[타입명]/_mapping?include_type_name=true&pretty' -H 'Content-Type:application/json' -d @[mapping json 파일]Mapping은 데이터 필드 타입을 지정하는 것으로 관계형 데이터 베이스의 Schema와 같다.
먼저 Mapping을 사용하기 위해 인덱스를 생성하고 조회해보자.
% curl -X PUT 'http://localhost:9200/classes?pretty' { "acknowledged" : true, "shards_acknowledged" : true, "index" : "classes" } % curl -X GET 'http://localhost:9200/classes?pretty' { "classes" : { "aliases" : { }, "mappings" : { }, "settings" : { "index" : { "routing" : { "allocation" : { "include" : { "_tier_preference" : "data_content" } } }, "number_of_shards" : "1", "provided_name" : "classes", "creation_date" : "1666922500930", "number_of_replicas" : "1", "uuid" : "23nwXW4BRmibLREDnk_yXA", "version" : { "created" : "7170499" } } } } }이제 mapping을 생성해서 적용하자. 먼저 mapping 파일을 생성해준다.
% vi classesRating_mapping.json { "class" : { "properties" : { "title" : { "type" : "string" }, "professor" : { "type" : "string" }, "major" : { "type" : "string" }, "semester" : { "type" : "string" }, "student_count" : { "type" : "integer" }, "unit" : { "type" : "integer" }, "rating" : { "type" : "integer" }, "submit_date" : { "type" : "date", "format" : "yyyy-MM-dd" }, "school_location" : { "type" : "geo_point" } } } }그런데 여기서 유튜브 영상대로 요청을 보내면 다음의 오류를 맞이하게 된다. elasticsearch 7.x 버젼부터는 curl 리퀘스트에서 헤더를 명확히 설정해주어야하고 또 mappign을 생성할 때에는 include_type_name을 true로 설정해주어야하기 때문이다.
참고한 사이트
↓
https://www.inflearn.com/questions/12385
elasticsearch 에러 관련입니다. - 인프런 | 질문 & 답변
elasticsearch 7.x 버젼부터는 curl 리퀘스트에서 헤더를 명확히 설정해주어야하고 또 mappign을 생성할 때에는 include_type_name을 true로 설정해주어야한다고 합니다. 이에 대한 에러문구는 아래와 같습니
www.inflearn.com
% curl -X PUT 'http://localhost:9200/classes/class/_mapping?pretty' -H 'Content-Type: application/json' -d @classesRating_mapping.json { "error" : { "root_cause" : [ { "type" : "illegal_argument_exception", "reason" : "Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true." } ], "type" : "illegal_argument_exception", "reason" : "Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true." }, "status" : 400 }또 여기서 다시 요청을 보내보면 다음의 오류를 응답 받는데 이는 작성해준 mapping.json파일에서 "string"부분을 "text"로 변경해주어야 함을 의미한다.
% curl -XPUT 'http://localhost:9200/classes/class/_mapping?include_type_name=true&pretty' -H 'Content-Type:application/json' -d @classesRating_mapping.json { "error" : { "root_cause" : [ { "type" : "mapper_parsing_exception", "reason" : "No handler for type [string] declared on field [professor]" } ], "type" : "mapper_parsing_exception", "reason" : "Failed to parse mapping [class]: No handler for type [string] declared on field [professor]", "caused_by" : { "type" : "mapper_parsing_exception", "reason" : "No handler for type [string] declared on field [professor]" } }, "status" : 400 }다시 classesRating_mapping.json 파일을 수정해주자.
% vi classesRating_mapping.json { "class" : { "properties" : { "title" : { "type" : "text" }, "professor" : { "type" : "text" }, "major" : { "type" : "text" }, "semester" : { "type" : "text" }, "student_count" : { "type" : "integer" }, "unit" : { "type" : "integer" }, "rating" : { "type" : "integer" }, "submit_date" : { "type" : "date", "format" : "yyyy-MM-dd" }, "school_location" : { "type" : "geo_point" } } } }다시 요청을 보내면 올바른 응답값을 받는 것을 확인할 수 있다.
% curl -XPUT 'http://localhost:9200/classes/class/_mapping?include_type_name=true&pretty' -H 'Content-Type:application/json' -d @classesRating_mapping.json { "acknowledged" : true }mapping이 잘 되었는지 GET을 통해 확인할 수 있다.
% curl -X GET 'http://localhost:9200/classes?pretty' { "classes" : { "aliases" : { }, "mappings" : { "properties" : { "major" : { "type" : "text" }, "professor" : { "type" : "text" }, "rating" : { "type" : "integer" }, "school_location" : { "type" : "geo_point" }, "semester" : { "type" : "text" }, "student_count" : { "type" : "integer" }, "submit_date" : { "type" : "date", "format" : "yyyy-MM-dd" }, "title" : { "type" : "text" }, "unit" : { "type" : "integer" } } }, "settings" : { "index" : { "routing" : { "allocation" : { "include" : { "_tier_preference" : "data_content" } } }, "number_of_shards" : "1", "provided_name" : "classes", "creation_date" : "1666922500930", "number_of_replicas" : "1", "uuid" : "23nwXW4BRmibLREDnk_yXA", "version" : { "created" : "7170499" } } } } }이번엔 bulk 데이터를 입력하고 타입에 잘 맞게 들어가는지 확인해보자. (위에서 사용한 데이터 다시 사용)
% curl -X POST 'http://localhost:9200/_bulk?pretty' -H 'Content-Type: application/json' --data-binary @classes.jsonGET을 통해 잘 들어간 것을 확이할 수 있다.
% curl -X GET 'http://localhost:9200/classes/class/5?pretty' { "_index" : "classes", "_type" : "class", "_id" : "5", "_version" : 1, "_seq_no" : 3, "_primary_term" : 1, "found" : true, "_source" : { "title" : "Machine Learning", "Professor" : "Tim Cook", "major" : "Computer Science", "semester" : [ "spring" ], "student_count" : 40, "unit" : 3, "rating" : 2, "submit_date" : "2016-04-02", "school_location" : { "lat" : 39.0, "lon" : -112.0 } } }Search
curl -X GET 'http://localhost:9200/[인덱스]/[타입명]/_search?pretty'이번엔 _search를 이용하여 데이터를 찾아보자. 먼저 bulk연산을 통해서 샘플 도큐먼트를 넣어준다.
% vi bulk_basketball.js { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "1" } } {"team" : "Golden States Warriors","name" : "Stephen Curry", "points" : 30,"rebounds" : 3,"assists" : 4, "blocks" : 5, "submit_date" : "2016-10-11"} { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "2" } } {"team" : "Golden States Warriors","name" : "Stephen Curry","points" : 32,"rebounds" : 5,"assists" : 8, "blocks" : 5, "submit_date" : "2016-10-13"} { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "3" } } {"team" : "Golden States Warriors","name" : "Stephen Curry","points" : 28,"rebounds" : 2,"assists" : 3, "blocks" : 1, "submit_date" : "2016-10-17"} { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "4" } } {"team" : "Golden States Warriors","name" : "Stephen Curry","points" : 36,"rebounds" : 1,"assists" : 2, "blocks" : 1, "submit_date" : "2016-11-20"} { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "5" } } {"team" : "Golden States Warriors","name" : "Stephen Curry","points" : 36,"rebounds" : 1,"assists" : 2, "blocks" : 1, "submit_date" : "2016-11-25"} { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "6" } } {"team" : "Golden States Warriors","name" : "Stephen Curry","points" : 32,"rebounds" : 1,"assists" : 4, "blocks" : 1, "submit_date" : "2016-11-28"} { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "7" } } {"team" : "Utah Jazz","name" : "Rudy Gobert", "points" : 3,"rebounds" : 11,"assists" : 4, "blocks" : 7, "submit_date" : "2016-10-12"} { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "8" } } {"team" : "Utah Jazz","name" : "Rudy Gobert","points" : 4,"rebounds" : 13,"assists" : 8, "blocks" : 5, "submit_date" : "2016-10-14"} { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "9" } } {"team" : "Utah Jazz","name" : "Rudy Gobert","points" : 8,"rebounds" : 10,"assists" : 3, "blocks" : 6, "submit_date" : "2016-10-18"} { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "10" } } {"team" : "Utah Jazz","name" : "Rudy Gobert","points" : 12,"rebounds" : 9,"assists" : 2, "blocks" : 6, "submit_date" : "2016-11-10"} { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "11" } } {"team" : "Utah Jazz","name" : "Rudy Gobert","points" : 12,"rebounds" : 14,"assists" : 2, "blocks" : 7, "submit_date" : "2016-11-22"} { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "12" } } {"team" : "Utah Jazz","name" : "Rudy Gobert","points" : 8,"rebounds" : 10,"assists" : 4, "blocks" : 5, "submit_date" : "2016-11-27"} { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "13" } } {"team" : "Washington Wizards","name" : "John Wall","points" : 8,"rebounds" : 1,"assists" : 13, "blocks" : 2, "submit_date" : "2016-10-18"} { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "14" } } {"team" : "Washington Wizards","name" : "John Wall","points" : 13,"rebounds" : 2,"assists" : 12, "blocks" : 3, "submit_date" : "2016-11-10"} { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "15" } } {"team" : "Washington Wizards","name" : "John Wall","points" : 15,"rebounds" : 3,"assists" : 12, "blocks" : 3, "submit_date" : "2016-11-22"} { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "16" } } {"team" : "Washington Wizards","name" : "John Wall","points" : 22,"rebounds" : 4,"assists" : 14, "blocks" : 3, "submit_date" : "2016-11-27"}% curl -X POST 'http://localhost:9200/_bulk?pretty' -H 'Content-Type: application/json' --data-binary @bulk_basketball.js { "took" : 961, "errors" : false, "items" : [ { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "1", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 0, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "2", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 1, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "3", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 2, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "4", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 3, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "5", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 4, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "6", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 5, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "7", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 6, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "8", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 7, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "9", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 8, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "10", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 9, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "11", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 10, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "12", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 11, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "13", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 12, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "14", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 13, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "15", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 14, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "16", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 15, "_primary_term" : 1, "status" : 201 } } ] }이번엔 _search연산을 통해서 basketball이라는 인덱스 안의 record 타입의 데이터를 모두 찾아보자.
% curl -X GET 'http://localhost:9200/basketball/record/_search?pretty' { "took" : 23, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 16, "relation" : "eq" }, "max_score" : 1.0, "hits" : [ { "_index" : "basketball", "_type" : "record", "_id" : "1", "_score" : 1.0, "_source" : { "team" : "Golden States Warriors", "name" : "Stephen Curry", "points" : 30, "rebounds" : 3, "assists" : 4, "blocks" : 5, "submit_date" : "2016-10-11" } }, { "_index" : "basketball", "_type" : "record", "_id" : "2", "_score" : 1.0, "_source" : { "team" : "Golden States Warriors", "name" : "Stephen Curry", "points" : 32, "rebounds" : 5, "assists" : 8, "blocks" : 5, "submit_date" : "2016-10-13" } }, { "_index" : "basketball", "_type" : "record", "_id" : "3", "_score" : 1.0, "_source" : { "team" : "Golden States Warriors", "name" : "Stephen Curry", "points" : 28, "rebounds" : 2, "assists" : 3, "blocks" : 1, "submit_date" : "2016-10-17" } }, { "_index" : "basketball", "_type" : "record", "_id" : "4", "_score" : 1.0, "_source" : { "team" : "Golden States Warriors", "name" : "Stephen Curry", "points" : 36, "rebounds" : 1, "assists" : 2, "blocks" : 1, "submit_date" : "2016-11-20" } }, { "_index" : "basketball", "_type" : "record", "_id" : "5", "_score" : 1.0, "_source" : { "team" : "Golden States Warriors", "name" : "Stephen Curry", "points" : 36, "rebounds" : 1, "assists" : 2, "blocks" : 1, "submit_date" : "2016-11-25" } }, { "_index" : "basketball", "_type" : "record", "_id" : "6", "_score" : 1.0, "_source" : { "team" : "Golden States Warriors", "name" : "Stephen Curry", "points" : 32, "rebounds" : 1, "assists" : 4, "blocks" : 1, "submit_date" : "2016-11-28" } }, { "_index" : "basketball", "_type" : "record", "_id" : "7", "_score" : 1.0, "_source" : { "team" : "Utah Jazz", "name" : "Rudy Gobert", "points" : 3, "rebounds" : 11, "assists" : 4, "blocks" : 7, "submit_date" : "2016-10-12" } }, { "_index" : "basketball", "_type" : "record", "_id" : "8", "_score" : 1.0, "_source" : { "team" : "Utah Jazz", "name" : "Rudy Gobert", "points" : 4, "rebounds" : 13, "assists" : 8, "blocks" : 5, "submit_date" : "2016-10-14" } }, { "_index" : "basketball", "_type" : "record", "_id" : "9", "_score" : 1.0, "_source" : { "team" : "Utah Jazz", "name" : "Rudy Gobert", "points" : 8, "rebounds" : 10, "assists" : 3, "blocks" : 6, "submit_date" : "2016-10-18" } }, { "_index" : "basketball", "_type" : "record", "_id" : "10", "_score" : 1.0, "_source" : { "team" : "Utah Jazz", "name" : "Rudy Gobert", "points" : 12, "rebounds" : 9, "assists" : 2, "blocks" : 6, "submit_date" : "2016-11-10" } } ] } }이번엔 쿼리를 통해서 points가 30인 기록만을 찾아보자.
curl -X GET 'http://localhost:9200/[인덱스]/[타입명]/_search?q=[필드]:[조건]&pretty'% curl -X GET 'http://localhost:9200/basketball/record/_search?q=points:30&pretty' { "took" : 31, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 1, "relation" : "eq" }, "max_score" : 1.0, "hits" : [ { "_index" : "basketball", "_type" : "record", "_id" : "1", "_score" : 1.0, "_source" : { "team" : "Golden States Warriors", "name" : "Stephen Curry", "points" : 30, "rebounds" : 3, "assists" : 4, "blocks" : 5, "submit_date" : "2016-10-11" } } ] } }쿼리문이 아닌 json 텍스트를 통해서 데이터를 찾을 수도 있다.
curl -X GET 'http://localhost:9200/[인덱스]/[타입]/_search?pretty' -H 'Content-Type: application/json' -d '[json 텍스트]'% curl -X GET 'http://localhost:9200/basketball/record/_search?pretty' -H 'Content-Type: application/json' -d '{"query":{"term":{"points":30}}}' { "took" : 14, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 1, "relation" : "eq" }, "max_score" : 1.0, "hits" : [ { "_index" : "basketball", "_type" : "record", "_id" : "1", "_score" : 1.0, "_source" : { "team" : "Golden States Warriors", "name" : "Stephen Curry", "points" : 30, "rebounds" : 3, "assists" : 4, "blocks" : 5, "submit_date" : "2016-10-11" } } ] } }Aggregation(metric)
curl -X GET 'http://localhost:9200/_search?pretty' -H 'Content-Type: application/json' --data-binary @[json 파일]이번엔 조합을 통해서 값을 도출하는 방법을 알아보자. 위에서 넣어놓은 basketball 도큐먼트를 활용한다.
- 평균 구하기
먼저 평균을 구하는 json 파일을 만들어준다.
% vi avg_points_aggs.json { "size" : 0, "aggs" : { "avg_score" : { "avg" : { "field" : "points" } } } }이를 이용해서 평균 구하기 요청을 보낸다.
% curl -X GET 'http://localhost:9200/_search?pretty' -H 'Content-Type: application/json' --data-binary @avg_points_aggs.json { "took" : 58, "timed_out" : false, "_shards" : { "total" : 2, "successful" : 2, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 40, "relation" : "eq" }, "max_score" : null, "hits" : [ ] }, "aggregations" : { "avg_score" : { "value" : 18.6875 } } }- max 구하기
이번엔 최대값을 구하는 json 파일을 만들어준다.
% vi max_points_aggs.json { "size" : 0, "aggs" : { "max_score" : { "max" : { "field" : "points" } } } }요청을 실행하면 최대값을 반환하는 것을 확인할 수 있다.
% curl -X GET 'http://localhost:9200/_search?pretty' -H 'Content-Type: application/json' --data-binary @max_points_aggs.json { "took" : 15, "timed_out" : false, "_shards" : { "total" : 2, "successful" : 2, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 40, "relation" : "eq" }, "max_score" : null, "hits" : [ ] }, "aggregations" : { "max_score" : { "value" : 36.0 } } }- min 구하기
최소값을 구하는 json 파일을 만들어준다.
% vi min_points_aggs.json { "size" : 0, "aggs" : { "min_score" : { "min" : { "field" : "points" } } } }요청을 실행하면 최소값을 반환하는 것을 확인할 수 있다.
% curl -X GET 'http://localhost:9200/_search?pretty' -H 'Content-Type: application/json' --data-binary @min_points_aggs.json { "took" : 23, "timed_out" : false, "_shards" : { "total" : 2, "successful" : 2, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 40, "relation" : "eq" }, "max_score" : null, "hits" : [ ] }, "aggregations" : { "min_score" : { "value" : 3.0 } } }- sum 구하기
합을 구하는 json 파일을 만들어준다.
% vi sum_points_aggs.json { "size" : 0, "aggs" : { "sum_score" : { "sum" : { "field" : "points" } } } }요청을 실행하면 합을 반환하는 것을 확인할 수 있다.
% curl -X GET 'http://localhost:9200/_search?pretty' -H 'Content-Type: application/json' --data-binary @sum_points_aggs.json { "took" : 14, "timed_out" : false, "_shards" : { "total" : 2, "successful" : 2, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 40, "relation" : "eq" }, "max_score" : null, "hits" : [ ] }, "aggregations" : { "sum_score" : { "value" : 299.0 } } }- stats 구하기
이번엔 위에서 구한 모든 값을 출력하는 stats을 이용해본다.
% vi stats_points.aggs.json { "size" : 0, "aggs" : { "stats_score" : { "stats" : { "field" : "points" } } } }요청을 보내면 위에서 구했던 평균, 최대, 최소, 합 등의 값을 반환하는 것을 확인할 수 있다.
% curl -X GET 'http://localhost:9200/_search?pretty' -H 'Content-Type: application/json' --data-binary @stats_points.aggs.json { "took" : 15, "timed_out" : false, "_shards" : { "total" : 2, "successful" : 2, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 40, "relation" : "eq" }, "max_score" : null, "hits" : [ ] }, "aggregations" : { "stats_score" : { "count" : 16, "min" : 3.0, "max" : 36.0, "avg" : 18.6875, "sum" : 299.0 } } }Aggregation(bucket)
이번엔 가지고 있는 document의 조합을 이용한 값을 반환하도록 하자.
먼저 위에서 사용했던 기존 데이터를 지우고 새로운 데이터를 넣어준 후 매핑해준다.
% curl -X DELETE 'http://localhost:9200/basketball' { "acknowledged":true } % vi twoteam_basketball.json { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "1" } } {"team" : "Chicago","name" : "Michael Jordan", "points" : 30,"rebounds" : 3,"assists" : 4, "blocks" : 3, "submit_date" : "1996-10-11"} { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "2" } } {"team" : "Chicago","name" : "Michael Jordan","points" : 20,"rebounds" : 5,"assists" : 8, "blocks" : 4, "submit_date" : "1996-10-13"} { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "3" } } {"team" : "LA","name" : "Kobe Bryant","points" : 30,"rebounds" : 2,"assists" : 8, "blocks" : 5, "submit_date" : "2014-10-13"} { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "4" } } {"team" : "LA","name" : "Kobe Bryant","points" : 40,"rebounds" : 4,"assists" : 8, "blocks" : 6, "submit_date" : "2014-11-13"} % curl -X POST 'http://localhost:9200/_bulk?pretty' -H 'Content-Type: application/json' --data-binary @twoteam_basketball.json { "took" : 907, "errors" : false, "items" : [ { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "1", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 0, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "2", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 1, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "3", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 2, "_primary_term" : 1, "status" : 201 } }, { "index" : { "_index" : "basketball", "_type" : "record", "_id" : "4", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 3, "_primary_term" : 1, "status" : 201 } } ] }이번엔 데이터를 이용해서 매핑해주자.
% vi basketball_mapping.json { "record" : { "properties" : { "team" : { "type" : "text", "fielddata" : true }, "name" : { "type" : "text", "fielddata" : true }, "points" : { "type" : "long" }, "rebounds" : { "type" : "long" }, "assists" : { "type" : "long" }, "blocks" : { "type" : "long" }, "submit_date" : { "type" : "date", "format" : "yyyy-MM-dd" } } } }먼저 위와 같은 json 파일로 요청을 보냈을 때 다음과 같은 응답값을 받게 되었다.
% curl -XPUT 'http://localhost:9200/basketball/record/_mapping?include_type_name=true&pretty' -H 'Content-Type:application/json' -d @basketball_mapping.json { "error" : { "root_cause" : [ { "type" : "illegal_argument_exception", "reason" : "Mapper for [submit_date] conflicts with existing mapper:\n\tCannot update parameter [format] from [strict_date_optional_time||epoch_millis] to [yyyy-MM-dd]" } ], "type" : "illegal_argument_exception", "reason" : "Mapper for [submit_date] conflicts with existing mapper:\n\tCannot update parameter [format] from [strict_date_optional_time||epoch_millis] to [yyyy-MM-dd]" }, "status" : 400 }자세히 보면 데이터 타입이 맞지 않아 생기는 오류로 기존 date 포맷을 "format" : "strict_date_optional_time||epoch_millis"로 변경해주어 해결해주었다.
% vi basketball_mapping.json { "record" : { "properties" : { "team" : { "type" : "text", "fielddata" : true }, "name" : { "type" : "text", "fielddata" : true }, "points" : { "type" : "long" }, "rebounds" : { "type" : "long" }, "assists" : { "type" : "long" }, "blocks" : { "type" : "long" }, "submit_date" : { "type" : "date", "format" : "strict_date_optional_time||epoch_millis" } } } }요청을 보내면 성공했다는 응답값을 받는다.
% curl -XPUT 'http://localhost:9200/basketball/record/_mapping?include_type_name=true&pretty' -H 'Content-Type:application/json' -d @basketball_mapping.json { "acknowledged" : true }다시한번 데이터를 조회해서 확인하면 데이터 타입이 잘 반영된 것을 확인할 수 있다.
% curl -X GET 'http://localhost:9200/basketball?pretty' { "basketball" : { "aliases" : { }, "mappings" : { "properties" : { "assists" : { "type" : "long" }, "blocks" : { "type" : "long" }, "name" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } }, "fielddata" : true }, "points" : { "type" : "long" }, "rebounds" : { "type" : "long" }, "submit_date" : { "type" : "date" }, "team" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } }, "fielddata" : true } } }, "settings" : { "index" : { "routing" : { "allocation" : { "include" : { "_tier_preference" : "data_content" } } }, "number_of_shards" : "1", "provided_name" : "basketball", "creation_date" : "1666925741390", "number_of_replicas" : "1", "uuid" : "EWUIScd3Sd2mLMInnV4KWQ", "version" : { "created" : "7170499" } } } } }이제 팀별로 그룹을 모으고 이를 통해 통계를 내는 것을 알아보자.현재 팀을 보면 다음과 같다.
통계를 내는 .json파일을 작성해준다.
% vi terms_aggs.json { "size" : 0, "aggs" : { "players" : { "terms" : { "field" : "team" } } } }이를 통해 요청을 보내면 팀별로 데이터를 받아오는 것을 확인할 수 있다.
% curl -XGET 'http://localhost:9200/_search?pretty' -H 'Content-Type:application/json' --data-binary @terms_aggs.json { "took" : 67, "timed_out" : false, "_shards" : { "total" : 2, "successful" : 2, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 28, "relation" : "eq" }, "max_score" : null, "hits" : [ ] }, "aggregations" : { "players" : { "doc_count_error_upper_bound" : 0, "sum_other_doc_count" : 0, "buckets" : [ { "key" : "chicago", "doc_count" : 2 }, { "key" : "la", "doc_count" : 2 } ] } } }이제 팀별로 document를 묶고 각 팀별로 데이터를 받아올 때 stats으로 받아오도록 설정한다.
% vi stats_by_team.json { "size" : 0, "aggs" : { "team_stats" : { "terms" : { "field" : "team" }, "aggs" : { "stats_score" : { "stats" : { "field" : "points" } } } } } }요청을 보내면 팀별로 통계를 잘 받아오는 것을 확인할 수 있다.
% curl -XGET 'http://localhost:9200/_search?pretty' -H 'Content-Type:application/json' --data-binary @stats_by_team.json { "took" : 20, "timed_out" : false, "_shards" : { "total" : 2, "successful" : 2, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 28, "relation" : "eq" }, "max_score" : null, "hits" : [ ] }, "aggregations" : { "team_stats" : { "doc_count_error_upper_bound" : 0, "sum_other_doc_count" : 0, "buckets" : [ { "key" : "chicago", "doc_count" : 2, "stats_score" : { "count" : 2, "min" : 20.0, "max" : 30.0, "avg" : 25.0, "sum" : 50.0 } }, { "key" : "la", "doc_count" : 2, "stats_score" : { "count" : 2, "min" : 30.0, "max" : 40.0, "avg" : 35.0, "sum" : 70.0 } } ] } } }( 참고한 사이트 ✨)
https://kim-dragon.tistory.com/20
ELK(ElasticSearch, Logstash, Kibana) 란? ELK Stack 이란?
ELK란? ELK는 3가지 오픈소스 소프트웨어 Elastic Search, LogStatsh, Kibana의 조합을 말합니다. 각 제품이 연동되어 데이터 수집 및 분석 툴로서 동작합니다. Elastic이라는 기업명에 걸맞게 높은 확장성과
kim-dragon.tistory.com
https://trandent.com/article/etc/detail/323366
Mac brew ELK (Elasticsearch + logstash + kibana) 설치 + mysql 연동 :: TRANDENT
Spring, JSP, Javascript, JQuery, AngularJS 등 웹개발 정보 공유. Trandent.com
trandent.com
https://logz.io/blog/brew-install-elasticsearch-mac/
Use Brew to Install Elasticsearch on Mac OS X | Logz.io
Use Homebrew to brew install Elasticsearch on Mac, along with the rest of the ELK Stack—Kibana, Filebeat, Metricbeat, and Logstash—with this tutorial
logz.io
https://velog.io/@gyudong/homebrew-ElasticSearch-Kibana-%EC%84%A4%EC%B9%98
[homebrew] ElasticSearch, Kibana 설치
ElasticSearch와 Kibana의 homebrew 설치 방법과 마주친 에러들, 에러 해결 방법까지
velog.io
'데이터베이스' 카테고리의 다른 글
ELK스택 기본사용법 - ③ Logstash + 실제 데이터 활용해보기 (1) 2022.10.31 ELK스택 기본사용법 - ② Kibana (0) 2022.10.31 동시성 제어 (0) 2022.08.18 조회 성능 개선하기 ( ③ DB 최적화, Replication ) (0) 2022.04.28 조회 성능 개선하기 ( ② 인덱스 설계 ) (0) 2022.04.28