Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Elasticsearch: Always use fixed_interval #50297

Merged
merged 1 commit into from Jun 13, 2022
Merged

Conversation

gabor
Copy link
Contributor

@gabor gabor commented Jun 7, 2022

Fixes #19911

in previous grafana versions, for Elasticsearch7.x we used the interval attribute to specify the interval-value. the problem is, this can become fixed_interval or calendar_interval based on the interval-value, so for example, when the user resized the dashboard-window, this might trigger a different interval-handling. this is not good, so we are switching to always use fixed_interval to provide consistent results. this is technically a breaking change, but it is a change to a better situation.

(NOTE: in elastic8, we already always use fixed_interval.)

how to test:

  • make devenv sources=elastic elastic_verison=7.10.0
  • make an elastic datasource that connect to this one where the elastic-version specified is 7.10. (simply copy the values from the gdev-elastic datasource, and change the elastic-version select-box). let's name it elastictest
  • you need a way to see the http-requests going from grafana to the elasticsearch-database
  • test frontend-part:
    • go to Explore, choose the elastictest datasource
    • a query will happen at this point, check the http-request that was sent to the elasticsearch-database, the request-params are in JSON, you should see the word "fixed_interval" there, and you should not see "interval" there
  • test backend-part:
    • create a dashboard, choose the elastictest datasource
    • add an alert, test the alert, check the http-request sent to the elasticsearch-database, the request-params are in JSON, you should see the word "fixed_interval" there, and you should not see "interval" there

Release notice breaking change

In Elasticsearch versions 7.x, to specify the interval-value we used the interval property. In Grafana 9.1.0 we switched to use the fixed_interval property. This makes it to be the same as in Elasticsearch versions 8.x, also this provides a more consistent experience, fixed_interval is a better match to Grafana's time invervals. For most situations this will not cause any visible change to query results.

@gabor gabor added add to changelog breaking change Relevant for changelog generation backport v9.0.x labels Jun 7, 2022
@gabor gabor added this to the 9.0.0 milestone Jun 7, 2022
@gabor gabor marked this pull request as ready for review June 7, 2022 10:28
@gabor gabor requested review from a team and Elfo404 and removed request for a team and Elfo404 June 7, 2022 10:28
@grafanabot
Copy link
Contributor

@grafanabot
Copy link
Contributor

@gabor-grafana gabor-grafana modified the milestones: 9.0.0, 9.1.0 Jun 7, 2022
Copy link
Member

@Elfo404 Elfo404 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

didn't test this but code looks good, makes sense to introduce this in 9.0, but i also think it's ok to wait for 9.1.

just one small nit about what i think are unneeded tests now

@@ -763,16 +763,12 @@ describe('ElasticQueryBuilder', () => {
extended_bounds: { max: '$timeTo', min: '$timeFrom' },
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since you are touching this file and we removed support for anything <7.10, i think it would be nice to change (at the top of this file this:

  const builder = new ElasticQueryBuilder({ timeField: '@timestamp', esVersion: '2.0.0' });
  const builder5x = new ElasticQueryBuilder({ timeField: '@timestamp', esVersion: '5.0.0' });
  const builder56 = new ElasticQueryBuilder({ timeField: '@timestamp', esVersion: '5.6.0' });
  const builder6x = new ElasticQueryBuilder({ timeField: '@timestamp', esVersion: '6.0.0' });
  const builder7x = new ElasticQueryBuilder({ timeField: '@timestamp', esVersion: '7.0.0' });
  const builder77 = new ElasticQueryBuilder({ timeField: '@timestamp', esVersion: '7.7.0' });
  const builder8 = new ElasticQueryBuilder({ timeField: '@timestamp', esVersion: '8.0.0' });

  const allBuilders = [builder, builder5x, builder56, builder6x, builder7x, builder77, builder8];

to

  const builder710 = new ElasticQueryBuilder({ timeField: '@timestamp', esVersion: '7.10.0' });
  const builder8 = new ElasticQueryBuilder({ timeField: '@timestamp', esVersion: '8.0.0' });

  const allBuilders = [builder710, builder8];

wdyt?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in general i agree, but i did the change and now got a bunch of errors below :) (some tests are done on builder, which is now gone, same for builder7x... i don't know right now if i should just delete the builder7x tests or adjust them etc.)

in short ,it seems some more work is needed in query_builder.test.ts, so if you're ok with it, i'd let it be for a later time.

@gabor gabor requested a review from a team June 7, 2022 11:23
@gabor gabor force-pushed the gabor/elastic-fixed-interval branch from 1d09b29 to 688199b Compare June 8, 2022 06:20
@gabor gabor added the no-backport Skip backport of PR label Jun 9, 2022
Copy link
Member

@ivanahuckova ivanahuckova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Great readme! Helped me a lot with testing this.

@rozingmichael
Copy link

Hi,
My team is experiencing this issue after upgrading our elastic to version 8.2.3. Is this fix released already in 9.0? (didn't find it mentioned in any release note) or is it delayed to 9.1? if so, when is this fix expected to be released?
Thanks.

@gabor
Copy link
Contributor Author

gabor commented Jul 21, 2022

hi, the fix will be released in grafana9.1.0 .

@linoman linoman modified the milestone: 9.1.0-beta1 Aug 2, 2022
@alimli
Copy link

alimli commented Aug 22, 2022

Hi,
I'm facing an issue with this change after upgrading from Grafana 9.0.3 to 9.1.0
Grafana: 9.1.0
ElasticSearch: 7.17.5

When setting a "Min Interval" value less than 365days, it is working as expected.
Screen Shot 2022-08-22 at 18 03 30

However if you sent a longer duration to "Min Interval" (e.g. 365d), the Interval is automatically converted to years.
Accordingly ES returns an error because units up to days are supported for fixed_interval as explained here.

Screen Shot 2022-08-22 at 18 03 49

Thanks in advance.

@gabor
Copy link
Contributor Author

gabor commented Aug 23, 2022

hi @alimli , could you please create this report as a separate github issue? it will make it easier for us to handle it. thanks!

@alimli
Copy link

alimli commented Aug 23, 2022

Hi @gabor, here it is.
#54075

@akosiaris
Copy link

Hi,

This broke one of my workflows, where I relied on calendar_interval to calculate some mean values based on actual calendar months (so 30d isn't a valid replacement). For now I reverted to 9.0.8, but I 'd like to ask if supporting calendar_interval is out of the question.

I 'd be fine with trying to write the code to support it (hopefully with a nice UI/UX), but before I dive in that rabbit hole (which per #19911 (comment) could be a deep one), it would be nice to know if the result would have any chance of being merged. Thanks!

@gabor
Copy link
Contributor Author

gabor commented Sep 12, 2022

hi @akosiaris , thank you for your feedback. before code starts being written, we should find out what exactly needs to be done, and how exactly the user-interface should behave. i created a feature-request discussion for it in #55034, if you could add your use-case, that would be great.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Elasticsearch 7.0 and interval deprecation, use fixed_interval or calendar_interval instead
9 participants