Elasticsearch does not modify the source, so you cannot change any part of
it when it is stored. You can store individual fields and retrieve their
value given the document. By default, elasticsearch does not store fields,
which the requires having the source to retrieve the field values. So
basically you can disable the source and enable storing only the fields you
want. Look at the store option:

Keep in mind that you would need to retrieve each field individually, which
means one seek per field, as opposed to one seek for the original document
source. For a few fields, the difference is negligible, but might be an
issue if you are using lots of stored fields.

Can you let me know, how to retrieve each field using JAVA API and also I
need to add some of the fields at runtime to _source using java API. If
possible can you please let me know, how to achieve it.

Elasticsearch does not modify the source, so you cannot change any part of
it when it is stored. You can store individual fields and retrieve their
value given the document. By default, elasticsearch does not store fields,
which the requires having the source to retrieve the field values. So
basically you can disable the source and enable storing only the fields you
want. Look at the store option:

Keep in mind that you would need to retrieve each field individually,
which means one seek per field, as opposed to one seek for the original
document source. For a few fields, the difference is negligible, but might
be an issue if you are using lots of stored fields.

In the future, please respond to the mailing list and not back to whoever
responded to you since there are others that can help and perhaps others in
the future with the same issue can read the replies. Also please note that
you sent three emails to the list and then two directly to me in a very
short time span. This action is incredibly rude. Please do not spam.

Can you let me know, how to retrieve each field using JAVA API and also I
need to add some of the fields at runtime to _source using java API. If
possible can you please let me know, how to achieve it.

Elasticsearch does not modify the source, so you cannot change any part
of it when it is stored. You can store individual fields and retrieve
their value given the document. By default, elasticsearch does not store
fields, which the requires having the source to retrieve the field values.
So basically you can disable the source and enable storing only the fields
you want. Look at the store option:

Keep in mind that you would need to retrieve each field individually,
which means one seek per field, as opposed to one seek for the original
document source. For a few fields, the difference is negligible, but might
be an issue if you are using lots of stored fields.

I have some data, which should be hidden/not displayed as part of the
_source.

when I set _source : { enabled : false}, source is being returned as
null.

How to make some fields are not displayed, but display _source.

Regards,
Akram.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

I was not properly aware of this UI, so I followed usual, gmail with reply
to Author button.

Sorry for Inconvenience.

Still I have some doubts on query posted.

When I set _source : {"enabled" : false}, I am not able to get value of
fields, If I try to get each field individually by
searchHit.getField("fieldName"). As I looked into the method getField it
internally fetching from source map only, where it is null.

In the future, please respond to the mailing list and not back to whoever
responded to you since there are others that can help and perhaps others in
the future with the same issue can read the replies. Also please note that
you sent three emails to the list and then two directly to me in a very
short time span. This action is incredibly rude. Please do not spam.

After retrieve the SearchHits from the SearchResponse object, you can use
the various field methods:

Can you let me know, how to retrieve each field using JAVA API and also I
need to add some of the fields at runtime to _source using java API. If
possible can you please let me know, how to achieve it.

Elasticsearch does not modify the source, so you cannot change any part
of it when it is stored. You can store individual fields and retrieve
their value given the document. By default, elasticsearch does not store
fields, which the requires having the source to retrieve the field values.
So basically you can disable the source and enable storing only the fields
you want. Look at the store option:

Keep in mind that you would need to retrieve each field individually,
which means one seek per field, as opposed to one seek for the original
document source. For a few fields, the difference is negligible, but might
be an issue if you are using lots of stored fields.

I have some data, which should be hidden/not displayed as part of the
_source.

when I set _source : { enabled : false}, source is being returned as
null.

How to make some fields are not displayed, but display _source.

Regards,
Akram.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

It would help if you provided your mapping and an example query. Are you
searching against a certain field, the data field, or the _all field?
Create a gist as described here: http://www.elasticsearch.org/help/ It
would help to retrieve the current mapping via the API, because some times
the mapping that is in effect might not be identical to the one you
supplied.

One issue to point out, in case it was not understood, is that some updates
to the mapping will not affect documents that are currently indexed. How
are you adding fields to a document? Are you reindexing or using the update
API?

One issue to point out, in case it was not understood, is that some
updates to the mapping will not affect documents that are currently
indexed. How are you adding fields to a document? Are you reindexing or
using the update API?

In your query code, you posted everything except the actual query! Which
field are you searching against? You mention that the data field is lost,
but you are excluding the text field. Typo?

I never exclude fields in source, but it seems like a field is required if
you want to search against it. It works after indexing, but will disappear
after updating. I doubt that is a bug, since elasticsearch has nothing to
update. An update basically is an atomic delete/insert.

One issue to point out, in case it was not understood, is that some
updates to the mapping will not affect documents that are currently
indexed. How are you adding fields to a document? Are you reindexing or
using the update API?

I have some data, which should be hidden/not displayed as part of the
_source.

when I set _source : { enabled : false}, source is being returned as
null.

How to make some fields are not displayed, but display _source.

Regards,
Akram.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

The missed line is the bold line, it will search on "data" field itself,
after updating/adding new fields to the source, the field "data" is getting
lost from source.

Let me know the possible ways to handle it.

Regards,
Akram.

On Friday, November 22, 2013 11:14:36 PM UTC+5:30, Ivan Brusic wrote:

In your query code, you posted everything except the actual query! Which
field are you searching against? You mention that the data field is lost,
but you are excluding the text field. Typo?

I never exclude fields in source, but it seems like a field is required if
you want to search against it. It works after indexing, but will disappear
after updating. I doubt that is a bug, since elasticsearch has nothing to
update. An update basically is an atomic delete/insert.

One issue to point out, in case it was not understood, is that some
updates to the mapping will not affect documents that are currently
indexed. How are you adding fields to a document? Are you reindexing or
using the update API?

I have some data, which should be hidden/not displayed as part of
the
_source.

when I set _source : { enabled : false}, source is being returned as
null.

How to make some fields are not displayed, but display _source.

Regards,
Akram.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

The missed line is the bold line, it will search on "data" field itself,
after updating/adding new fields to the source, the field "data" is getting
lost from source.

Let me know the possible ways to handle it.

Regards,
Akram.

On Friday, November 22, 2013 11:14:36 PM UTC+5:30, Ivan Brusic wrote:

In your query code, you posted everything except the actual query! Which
field are you searching against? You mention that the data field is lost,
but you are excluding the text field. Typo?

I never exclude fields in source, but it seems like a field is required
if you want to search against it. It works after indexing, but will
disappear after updating. I doubt that is a bug, since elasticsearch has
nothing to update. An update basically is an atomic delete/insert.

One issue to point out, in case it was not understood, is that some
updates to the mapping will not affect documents that are currently
indexed. How are you adding fields to a document? Are you reindexing or
using the update API?

I have some data, which should be hidden/not displayed as part of
the
_source.

when I set _source : { enabled : false}, source is being returned
as null.

How to make some fields are not displayed, but display _source.

Regards,
Akram.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

The missed line is the bold line, it will search on "data" field itself,
after updating/adding new fields to the source, the field "data" is getting
lost from source.

Let me know the possible ways to handle it.

Regards,
Akram.

On Friday, November 22, 2013 11:14:36 PM UTC+5:30, Ivan Brusic wrote:

In your query code, you posted everything except the actual query! Which
field are you searching against? You mention that the data field is lost,
but you are excluding the text field. Typo?

I never exclude fields in source, but it seems like a field is required
if you want to search against it. It works after indexing, but will
disappear after updating. I doubt that is a bug, since elasticsearch has
nothing to update. An update basically is an atomic delete/insert.

One issue to point out, in case it was not understood, is that some
updates to the mapping will not affect documents that are currently
indexed. How are you adding fields to a document? Are you reindexing or
using the update API?

I have some data, which should be hidden/not displayed as part of
the
_source.

when I set _source : { enabled : false}, source is being returned
as null.

How to make some fields are not displayed, but display _source.

Regards,
Akram.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.

An update in Lucene is basically an atomic delete and update. Elasticsearch
needs the source if it is to preserve the content between updates. I would
argue that the real bug here is that elasticsearch should not have indexed
the excluded field in the first place.

Perhaps the better question is what are you trying to solve by excluding
the field? Maybe you can leave the source intact and solve the original
issue in a different manner.

The missed line is the bold line, it will search on "data" field itself,
after updating/adding new fields to the source, the field "data" is getting
lost from source.

Let me know the possible ways to handle it.

Regards,
Akram.

On Friday, November 22, 2013 11:14:36 PM UTC+5:30, Ivan Brusic wrote:

In your query code, you posted everything except the actual query!
Which field are you searching against? You mention that the data field is
lost, but you are excluding the text field. Typo?

I never exclude fields in source, but it seems like a field is required
if you want to search against it. It works after indexing, but will
disappear after updating. I doubt that is a bug, since elasticsearch has
nothing to update. An update basically is an atomic delete/insert.

One issue to point out, in case it was not understood, is that some
updates to the mapping will not affect documents that are currently
indexed. How are you adding fields to a document? Are you reindexing or
using the update API?

I have some data, which should be hidden/not displayed as part of
the
_source.

when I set _source : { enabled : false}, source is being returned
as null.

How to make some fields are not displayed, but display _source.

Regards,
Akram.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.

As the "data" field is sensitive data, customer don't want it to be visible
as part of the source. But it should be searchable.

Let me know, if any way to achieve this.

Regards,
Akram.

On Friday, November 22, 2013 11:59:16 PM UTC+5:30, Ivan Brusic wrote:

Yes, I did ask before if that was a typo.

An update in Lucene is basically an atomic delete and update.
Elasticsearch needs the source if it is to preserve the content between
updates. I would argue that the real bug here is that elasticsearch should
not have indexed the excluded field in the first place.

Perhaps the better question is what are you trying to solve by excluding
the field? Maybe you can leave the source intact and solve the original
issue in a different manner.

The missed line is the bold line, it will search on "data" field
itself, after updating/adding new fields to the source, the field "data" is
getting lost from source.

Let me know the possible ways to handle it.

Regards,
Akram.

On Friday, November 22, 2013 11:14:36 PM UTC+5:30, Ivan Brusic wrote:

In your query code, you posted everything except the actual query!
Which field are you searching against? You mention that the data field is
lost, but you are excluding the text field. Typo?

I never exclude fields in source, but it seems like a field is
required if you want to search against it. It works after indexing, but
will disappear after updating. I doubt that is a bug, since elasticsearch
has nothing to update. An update basically is an atomic delete/insert.

One issue to point out, in case it was not understood, is that some
updates to the mapping will not affect documents that are currently
indexed. How are you adding fields to a document? Are you reindexing or
using the update API?

I have some data, which should be hidden/not displayed as part
of the
_source.

when I set _source : { enabled : false}, source is being
returned as null.

How to make some fields are not displayed, but display _source.

Regards,
Akram.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.

Perhaps others will chime in, but off the top of my head you have two
options:

Fix the data in the application side before it is returned to the actual
client. Are you returning straight JSON back to the client? Use Jackson
(which is what elasticsearch also uses) to transform the JSON response.

Use fields and not the source. Each field is indexed, but not stored by
default. Explicitly store the fields you want and retrieve them.

However, as you demonstrated, source needs to be enabled if you want to use
the update API. If you can skip the update API and simply reindex, the
issue is no longer relevant. If not, post-processing the result would be
your best bet. Maybe others would have some ideas.

As the "data" field is sensitive data, customer don't want it to be
visible as part of the source. But it should be searchable.

Let me know, if any way to achieve this.

Regards,
Akram.

On Friday, November 22, 2013 11:59:16 PM UTC+5:30, Ivan Brusic wrote:

Yes, I did ask before if that was a typo.

An update in Lucene is basically an atomic delete and update.
Elasticsearch needs the source if it is to preserve the content between
updates. I would argue that the real bug here is that elasticsearch should
not have indexed the excluded field in the first place.

Perhaps the better question is what are you trying to solve by excluding
the field? Maybe you can leave the source intact and solve the original
issue in a different manner.

The missed line is the bold line, it will search on "data" field
itself, after updating/adding new fields to the source, the field "data" is
getting lost from source.

Let me know the possible ways to handle it.

Regards,
Akram.

On Friday, November 22, 2013 11:14:36 PM UTC+5:30, Ivan Brusic wrote:

In your query code, you posted everything except the actual query!
Which field are you searching against? You mention that the data field is
lost, but you are excluding the text field. Typo?

I never exclude fields in source, but it seems like a field is
required if you want to search against it. It works after indexing, but
will disappear after updating. I doubt that is a bug, since elasticsearch
has nothing to update. An update basically is an atomic delete/insert.

One issue to point out, in case it was not understood, is that some
updates to the mapping will not affect documents that are currently
indexed. How are you adding fields to a document? Are you reindexing or
using the update API?

I have some data, which should be hidden/not displayed as part
of the
_source.

when I set _source : { enabled : false}, source is being
returned as null.

How to make some fields are not displayed, but display _source.

Regards,
Akram.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.

Perhaps others will chime in, but off the top of my head you have two
options:

Fix the data in the application side before it is returned to the
actual client. Are you returning straight JSON back to the client? Use
Jackson (which is what elasticsearch also uses) to transform the JSON
response.

Use fields and not the source. Each field is indexed, but not stored by
default. Explicitly store the fields you want and retrieve them.

However, as you demonstrated, source needs to be enabled if you want to
use the update API. If you can skip the update API and simply reindex, the
issue is no longer relevant. If not, post-processing the result would be
your best bet. Maybe others would have some ideas.

As the "data" field is sensitive data, customer don't want it to be
visible as part of the source. But it should be searchable.

Let me know, if any way to achieve this.

Regards,
Akram.

On Friday, November 22, 2013 11:59:16 PM UTC+5:30, Ivan Brusic wrote:

Yes, I did ask before if that was a typo.

An update in Lucene is basically an atomic delete and update.
Elasticsearch needs the source if it is to preserve the content between
updates. I would argue that the real bug here is that elasticsearch should
not have indexed the excluded field in the first place.

Perhaps the better question is what are you trying to solve by excluding
the field? Maybe you can leave the source intact and solve the original
issue in a different manner.

The missed line is the bold line, it will search on "data" field
itself, after updating/adding new fields to the source, the field "data" is
getting lost from source.

Let me know the possible ways to handle it.

Regards,
Akram.

On Friday, November 22, 2013 11:14:36 PM UTC+5:30, Ivan Brusic wrote:

In your query code, you posted everything except the actual query!
Which field are you searching against? You mention that the data field is
lost, but you are excluding the text field. Typo?

I never exclude fields in source, but it seems like a field is
required if you want to search against it. It works after indexing, but
will disappear after updating. I doubt that is a bug, since elasticsearch
has nothing to update. An update basically is an atomic delete/insert.

One issue to point out, in case it was not understood, is that
some updates to the mapping will not affect documents that are currently
indexed. How are you adding fields to a document? Are you reindexing or
using the update API?

I have some data, which should be hidden/not displayed as part
of the
_source.

when I set _source : { enabled : false}, source is being
returned as null.

How to make some fields are not displayed, but display
_source.

Regards,
Akram.

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.

If you don't expose the raw Elasticsearch API to your customers, then
you can remove the sensitive information from the source JSON in your
own application logic before you publish it. You may want to use a
library like Jackson to work with JSON data:https://github.com/FasterXML/jackson

Use separate fields and not the source. This way was described in
Ivans other posts.

But I fear it may not help you, if you want to be able to use the update
API without having to provide the sensitive information again. If this
information is not stored, due to exclusion from the source field or due
to a non-stored field, then you will have to provide it again when
updating it. All fields of a document have to be reindexed when
updating. If they are stored or part of the source they will be reused.
If they are not, they will be not be part of the new document, unless
they are provided with in the update.

Perhaps others will chime in, but off the top of my head you have two
options:

Fix the data in the application side before it is returned to the
actual client. Are you returning straight JSON back to the client? Use
Jackson (which is what elasticsearch also uses) to transform the JSON
response.

Use fields and not the source. Each field is indexed, but not stored by
default. Explicitly store the fields you want and retrieve them.

However, as you demonstrated, source needs to be enabled if you want to
use the update API. If you can skip the update API and simply reindex, the
issue is no longer relevant. If not, post-processing the result would be
your best bet. Maybe others would have some ideas.

As the "data" field is sensitive data, customer don't want it to be
visible as part of the source. But it should be searchable.

Let me know, if any way to achieve this.

Regards,
Akram.

On Friday, November 22, 2013 11:59:16 PM UTC+5:30, Ivan Brusic wrote:

Yes, I did ask before if that was a typo.

An update in Lucene is basically an atomic delete and update.
Elasticsearch needs the source if it is to preserve the content between
updates. I would argue that the real bug here is that elasticsearch should
not have indexed the excluded field in the first place.

Perhaps the better question is what are you trying to solve by excluding
the field? Maybe you can leave the source intact and solve the original
issue in a different manner.

The missed line is the bold line, it will search on "data" field
itself, after updating/adding new fields to the source, the field "data" is
getting lost from source.

Let me know the possible ways to handle it.

Regards,
Akram.

On Friday, November 22, 2013 11:14:36 PM UTC+5:30, Ivan Brusic wrote:

In your query code, you posted everything except the actual query!
Which field are you searching against? You mention that the data field is
lost, but you are excluding the text field. Typo?

I never exclude fields in source, but it seems like a field is
required if you want to search against it. It works after indexing, but
will disappear after updating. I doubt that is a bug, since elasticsearch
has nothing to update. An update basically is an atomic delete/insert.

One issue to point out, in case it was not understood, is that
some updates to the mapping will not affect documents that are currently
indexed. How are you adding fields to a document? Are you reindexing or
using the update API?

I have some data, which should be hidden/not displayed as part
of the
_source.

when I set _source : { enabled : false}, source is being
returned as null.

How to make some fields are not displayed, but display
_source.

Regards,
Akram.

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.