Hello community,
here is the log from the commit of package rubygem-fluentd for openSUSE:Factory checked in at 2019-04-01 12:37:12
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Comparing /work/SRC/openSUSE:Factory/rubygem-fluentd (Old)
and /work/SRC/openSUSE:Factory/.rubygem-fluentd.new.25356 (New)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "rubygem-fluentd"
Mon Apr 1 12:37:12 2019 rev:10 rq:689713 version:1.4.1
Changes:
--------
--- /work/SRC/openSUSE:Factory/rubygem-fluentd/rubygem-fluentd.changes 2019-01-21 10:54:07.575745147 +0100
+++ /work/SRC/openSUSE:Factory/.rubygem-fluentd.new.25356/rubygem-fluentd.changes 2019-04-01 12:37:12.805881247 +0200
@@ -1,0 +2,12 @@
+Fri Mar 29 06:00:25 UTC 2019 - Stephan Kulow
+
+- updated to version 1.4.1
+ see installed CHANGELOG.md
+
+-------------------------------------------------------------------
+Sat Mar 2 15:14:08 UTC 2019 - Stephan Kulow
+
+- updated to version 1.4.0
+ see installed CHANGELOG.md
+
+-------------------------------------------------------------------
Old:
----
fluentd-1.3.3.gem
New:
----
fluentd-1.4.1.gem
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Other differences:
------------------
++++++ rubygem-fluentd.spec ++++++
--- /var/tmp/diff_new_pack.eToZ38/_old 2019-04-01 12:37:13.357881518 +0200
+++ /var/tmp/diff_new_pack.eToZ38/_new 2019-04-01 12:37:13.361881520 +0200
@@ -24,7 +24,7 @@
#
Name: rubygem-fluentd
-Version: 1.3.3
+Version: 1.4.1
Release: 0
%define mod_name fluentd
%define mod_full_name %{mod_name}-%{version}
++++++ fluentd-1.3.3.gem -> fluentd-1.4.1.gem ++++++
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/.github/ISSUE_TEMPLATE.md new/.github/ISSUE_TEMPLATE.md
--- old/.github/ISSUE_TEMPLATE.md 2019-01-07 04:07:42.000000000 +0100
+++ new/.github/ISSUE_TEMPLATE.md 2019-03-19 14:48:00.000000000 +0100
@@ -1,6 +1,8 @@
Check [CONTRIBUTING guideline](https://github.com/fluent/fluentd/blob/master/CONTRIBUTING.md) first and here is the list to help us investigate the problem.
- fluentd or td-agent version.
-- Environment information, e.g. OS.
+- Environment information:
+ - Operating system: `cat /etc/os-release`
+ - Kernel version: `uname -r`
- Your configuration
- Your problem explanation. If you have an error logs, write it together.
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/.github/PULL_REQUEST_TEMPLATE.md new/.github/PULL_REQUEST_TEMPLATE.md
--- old/.github/PULL_REQUEST_TEMPLATE.md 1970-01-01 01:00:00.000000000 +0100
+++ new/.github/PULL_REQUEST_TEMPLATE.md 2019-03-19 14:48:00.000000000 +0100
@@ -0,0 +1,13 @@
+<!--
+Thank you for contributing to Fluentd!
+Please provide the following information to help us make the most of your pull request:
+-->
+
+**Which issue(s) this PR fixes**:
+Fixes #
+
+**What this PR does / why we need it**:
+
+**Docs Changes**:
+
+**Release Note**:
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/.gitignore new/.gitignore
--- old/.gitignore 2019-01-07 04:07:42.000000000 +0100
+++ new/.gitignore 2019-03-19 14:48:00.000000000 +0100
@@ -25,4 +25,5 @@
coverage/*
.vagrant/
cov-int/
-cov-fluentd.tar.gz
\ No newline at end of file
+cov-fluentd.tar.gz
+.vscode
\ No newline at end of file
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/.travis.yml new/.travis.yml
--- old/.travis.yml 2019-01-07 04:07:42.000000000 +0100
+++ new/.travis.yml 2019-03-19 14:48:00.000000000 +0100
@@ -11,8 +11,6 @@
os: linux
- rvm: 2.2.10
os: linux
- - rvm: 2.3.7
- os: linux
- rvm: 2.4.5
os: linux
- rvm: 2.5.3
@@ -37,6 +35,8 @@
os: osx
osx_image: xcode8.3 # OSX 10.12
allow_failures:
+ - rvm: 2.3.8
+ os: linux
- rvm: 2.1.10
os: osx
osx_image: xcode8.3
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/CHANGELOG.md new/CHANGELOG.md
--- old/CHANGELOG.md 2019-01-07 04:07:42.000000000 +0100
+++ new/CHANGELOG.md 2019-03-19 14:48:00.000000000 +0100
@@ -1,6 +1,55 @@
+# v1.4
+
+## Release v1.4.1 - 2019/03/18
+
+### Enhancements
+
+* system: Add worker_id to process_name when workers is larger than 1
+ https://github.com/fluent/fluentd/pull/2321
+* in_syslog: Check message length when read from buffer in octet counting
+ https://github.com/fluent/fluentd/pull/2323
+* parser_regexp: Check named captures. When no named captures, configuration error is raised
+ https://github.com/fluent/fluentd/pull/2331
+
+### Bug fixes
+
+* out_forward: Make tls_client_private_key_passphrase secret
+ https://github.com/fluent/fluentd/pull/2324
+
+## Release v1.4.0 - 2019/02/24
+
+### New features
+
+* multiprocess: Support <worker N-M> syntax
+ https://github.com/fluent/fluentd/pull/2292
+* output: Work <secondary> and retry_forever together
+ https://github.com/fluent/fluentd/pull/2276
+* out_file: Support placeholders in symlink_path
+ https://github.com/fluent/fluentd/pull/2254
+
+### Enhancements
+
+* output: Add MessagePack unpacker error to unrecoverable error list
+ https://github.com/fluent/fluentd/pull/2301
+* output: Reduce flush delay when large timekey and small timekey_wait are specified
+ https://github.com/fluent/fluentd/pull/2291
+* config: Support embedded ruby code in section argument.
+ https://github.com/fluent/fluentd/pull/2295
+* in_tail: Improve encoding parameter handling
+ https://github.com/fluent/fluentd/pull/2305
+* in_tcp/in_udp: Add <parse> section check
+ https://github.com/fluent/fluentd/pull/2267
+
+### Bug fixes
+
+* server: Ignore IOError and related errors in UDP
+ https://github.com/fluent/fluentd/pull/2310
+* server: Ignore EPIPE in TLS accept to avoid fluentd restart
+ https://github.com/fluent/fluentd/pull/2253
+
# v1.3
-## Release v1.3.3 - 2018/01/06
+## Release v1.3.3 - 2019/01/06
### Enhancements
@@ -642,12 +691,12 @@
* output: Secondary calculation should consider 'retry_max_times'
https://github.com/fluent/fluentd/pull/1452
-* Fix regression of deprecatd 'process' module
+* Fix regression of deprecated 'process' module
https://github.com/fluent/fluentd/pull/1443
* Fix missing parser_regex require
https://github.com/fluent/fluentd/issues/1458
https://github.com/fluent/fluentd/pull/1453
-* Keep 'Fluent::BufferQueueLimitError' for exsting plugins
+* Keep 'Fluent::BufferQueueLimitError' for existing plugins
https://github.com/fluent/fluentd/pull/1456
* in_tail: Untracked files should be removed from watching list to avoid memory bloat
https://github.com/fluent/fluentd/pull/1467
@@ -899,7 +948,7 @@
### Bug fixes
-* Raise configuration errors to clearify what's wrong when "@type" is missing
+* Raise configuration errors to clarify what's wrong when "@type" is missing
https://github.com/fluent/fluentd/pull/1202
* Fix the bug not to launch Fluentd when v0.12 MultiOutput plugin is configured
https://github.com/fluent/fluentd/pull/1206
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/CONTRIBUTING.md new/CONTRIBUTING.md
--- old/CONTRIBUTING.md 2019-01-07 04:07:42.000000000 +0100
+++ new/CONTRIBUTING.md 2019-03-19 14:48:00.000000000 +0100
@@ -27,7 +27,7 @@
* **Documentation**: Use [fluentd-docs](https://github.com/fluent/fluentd-docs) repository.
If you find a bug of 3rd party plugins, please submit an issue to each plugin repository.
-And use [omnibus-td-agent](https://github.com/treasure-data/omnibus-td-agent) repository for td-agent releated issues.
+And use [omnibus-td-agent](https://github.com/treasure-data/omnibus-td-agent) repository for td-agent related issues.
Note: Before report the issue, check latest version first. Sometimes users report fixed bug with older version.
@@ -37,7 +37,7 @@
* Write tests.
* Run tests before send Pull Request by `bundle exec rake test`
-* Write a [good commit message](http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html).
+* Write a [good commit message](https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html).
* Fluentd repositories needs [DCO](https://github.com/apps/dco) on PR. Please add `Signed-off-by` to the commit(See DCO link for more detail).
There are some patches which are hard to write tests, e.g. process handling, concurrency issue or etc.
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/GithubWorkflow.md new/GithubWorkflow.md
--- old/GithubWorkflow.md 1970-01-01 01:00:00.000000000 +0100
+++ new/GithubWorkflow.md 2019-03-19 14:48:00.000000000 +0100
@@ -0,0 +1,78 @@
+# Github workflow for contributing to fluentd
+
+Table of Contents
+
+* [Fork a repository](#fork-a-repository)
+* [Clone fork repository to local](#clone-fork-repository-to-local)
+* [Create a branch to add a new feature or fix issues](#create-a-branch-to-add-a-new-feature-or-fix-issues)
+* [Commit and Push](#commit-and-push)
+* [Create a Pull Request](#create-a-pull-request)
+
+
+The [fluentd](https://github.com/fluent/fluentd.git) code is hosted on Github (https://github.com/fluent/fluentd). The repository is called `upstream`. Contributors will develop and commit their changes in a clone of upstream repository. Then contributors push their change to their forked repository (`origin`) and create a Pull Request (PR), the PR will be merged to `upstream` repository if it meets the all the necessary requirements.
+
+## Fork a repository
+
+ Go to https://github.com/fluent/fluentd then hit the `Fork` button to fork your own copy of repository **fluentd** to your github account.
+
+## Clone the forked repository to local
+
+Clone the forked repo in [above step](#fork-a-repository) to your local working directory:
+```sh
+$ git clone https://github.com/$your_github_account/fluentd.git
+```
+
+Keep your fork in sync with the main repo, add an `upstream` remote:
+```sh
+$ cd fluentd
+$ git remote add upstream https://github.com/fluentd/fluentd.git
+$ git remote -v
+
+origin https://github.com/$your_github_account/fluentd.git (fetch)
+origin https://github.com/$your_github_account/fluentd.git (push)
+upstream https://github.com/fluentd/fluentd.git (fetch)
+upstream https://github.com/fluentd/fluentd.git (push)
+```
+
+Sync your local `master` branch:
+```sh
+$ git checkout master
+$ git pull origin master
+$ git fetch upstream
+$ git rebase upstream/master
+```
+
+## Create a branch to add a new feature or fix issues
+
+Before making any change, create a new branch:
+```sh
+$ git checkout master
+$ git pull upstream master
+$ git checkout -b new-feature
+```
+
+## Commit and Push
+
+Make any change on the branch `new-feature` then build and test your codes.
+Include in what will be committed:
+```sh
+$ git add <file>
+```
+
+Commit your changes with `sign-offs`
+```sh
+$ git commit -s
+```
+
+Enter your commit message to describe the changes. See the tips for a good commit message at [here](https://chris.beams.io/posts/git-commit/).
+Likely you go back and edit/build/test some more then `git commit --amend`
+
+Push your branch `new-feature` to your forked repository:
+```sh
+$ git push -u origin new-feature
+```
+
+## Create a Pull Request
+
+* Go to your fork at https://github.com/$your_github_account/fluentd
+* Create a Pull Request from the branch you recently pushed by hitting the button `Compare & pull request` next to branch name.
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/MAINTAINERS.md new/MAINTAINERS.md
--- old/MAINTAINERS.md 2019-01-07 04:07:42.000000000 +0100
+++ new/MAINTAINERS.md 2019-03-19 14:48:00.000000000 +0100
@@ -1,6 +1,6 @@
# Fluentd Maintainers
-- [Naotoshi Seo](https://github.com/sonots), [DeNA](http://dena.com/intl/)
+- [Naotoshi Seo](https://github.com/sonots), [ZOZO Technologies](https://tech.zozo.com/en/)
- [Okkez](https://github.com/okkez), [Clearcode](https://www.clear-code.com/)
- [Hiroshi Hatake](https://github.com/cosmo0920), [Clearcode](https://www.clear-code.com/)
- [Masahiro Nakagawa](https://github.com/repeatedly), [Treasure Data](https://www.treasuredata.com/)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/README.md new/README.md
--- old/README.md 2019-01-07 04:07:42.000000000 +0100
+++ new/README.md 2019-03-19 14:48:00.000000000 +0100
@@ -71,8 +71,8 @@
- Project repository: https://github.com/fluent
- Discussion: https://groups.google.com/group/fluentd
- Slack / Community: https://slack.fluentd.org
-- Newsletters: https://www.fluentd.org/newsletter_signup
-- Author: Sadayuki Furuhashi
+- Newsletters: https://www.fluentd.org/newsletter
+- Author: [Sadayuki Furuhashi](https://github.com/frsyuki)
- Copyright: 2011-2018 Fluentd Authors
- License: Apache License, Version 2.0
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/appveyor.yml new/appveyor.yml
--- old/appveyor.yml 2019-01-07 04:07:42.000000000 +0100
+++ new/appveyor.yml 2019-03-19 14:48:00.000000000 +0100
@@ -30,14 +30,9 @@
devkit: C:\Ruby23\DevKit
- ruby_version: "22-x64"
devkit: C:\Ruby23-x64\DevKit
- - ruby_version: "21-x64"
- devkit: C:\Ruby23-x64\DevKit
- ruby_version: "22"
devkit: C:\Ruby23\DevKit
WIN_RAPID: true
- - ruby_version: "21"
- devkit: C:\Ruby23\DevKit
- WIN_RAPID: true
matrix:
allow_failures:
- ruby_version: "21"
Binary files old/checksums.yaml.gz and new/checksums.yaml.gz differ
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/command/plugin_config_formatter.rb new/lib/fluent/command/plugin_config_formatter.rb
--- old/lib/fluent/command/plugin_config_formatter.rb 2019-01-07 04:07:42.000000000 +0100
+++ new/lib/fluent/command/plugin_config_formatter.rb 2019-03-19 14:48:00.000000000 +0100
@@ -32,7 +32,8 @@
"buffer", "parser", "formatter", "storage"
]
- DOCS_BASE_URL = "https://docs.fluentd.org/v1.0/articles/"
+ DOCS_BASE_URL = "https://docs.fluentd.org/v1.0/articles/quickstart"
+ DOCS_ARTICLE_BASE_URL = "https://docs.fluentd.org/v1.0/articles/"
def initialize(argv = ARGV)
@argv = argv
@@ -185,7 +186,7 @@
end
def plugin_helper_url(plugin_helper)
- "#{DOCS_BASE_URL}api-plugin-helper-#{plugin_helper}"
+ "#{DOCS_ARTICLE_BASE_URL}api-plugin-helper-#{plugin_helper}"
end
def plugin_helper_markdown_link(plugin_helper)
@@ -194,7 +195,7 @@
def plugin_overview_url(class_name)
plugin_type = class_name.slice(/::(\w+)\z/, 1).downcase
- "#{DOCS_BASE_URL}#{plugin_type}-plugin-overview"
+ "#{DOCS_ARTICLE_BASE_URL}#{plugin_type}-plugin-overview"
end
def plugin_overview_markdown_link(class_name)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/config/element.rb new/lib/fluent/config/element.rb
--- old/lib/fluent/config/element.rb 2019-01-07 04:07:42.000000000 +0100
+++ new/lib/fluent/config/element.rb 2019-03-19 14:48:00.000000000 +0100
@@ -36,12 +36,12 @@
# it's global logger, not plugin logger: deprecated message should be global warning, not plugin level.
@logger = defined?($log) ? $log : nil
- @target_worker_id = nil
+ @target_worker_ids = []
end
attr_accessor :name, :arg, :unused, :v1_config, :corresponding_proxies, :unused_in
attr_writer :elements
- attr_reader :target_worker_id
+ attr_reader :target_worker_ids
RESERVED_PARAMETERS_COMPAT = {
'@type' => 'type',
@@ -223,22 +223,29 @@
end
def set_target_worker_id(worker_id)
- @target_worker_id = worker_id
+ @target_worker_ids = [worker_id]
@elements.each { |e|
e.set_target_worker_id(worker_id)
}
end
+ def set_target_worker_ids(worker_ids)
+ @target_worker_ids = worker_ids.uniq
+ @elements.each { |e|
+ e.set_target_worker_ids(worker_ids.uniq)
+ }
+ end
+
def for_every_workers?
- @target_worker_id == nil
+ @target_worker_ids.empty?
end
def for_this_worker?
- @target_worker_id == Fluent::Engine.worker_id
+ @target_worker_ids.include?(Fluent::Engine.worker_id)
end
def for_another_worker?
- @target_worker_id != nil && @target_worker_id != Fluent::Engine.worker_id
+ !@target_worker_ids.empty? && !@target_worker_ids.include?(Fluent::Engine.worker_id)
end
end
end
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/config/v1_parser.rb new/lib/fluent/config/v1_parser.rb
--- old/lib/fluent/config/v1_parser.rb 2019-01-07 04:07:42.000000000 +0100
+++ new/lib/fluent/config/v1_parser.rb 2019-03-19 14:48:00.000000000 +0100
@@ -82,7 +82,7 @@
elsif skip(/\)
e_name = scan(ELEMENT_NAME)
spacing
- e_arg = scan_nonquoted_string(/(?:#{ZERO_OR_MORE_SPACING}\>)/)
+ e_arg = scan_string(/(?:#{ZERO_OR_MORE_SPACING}\>)/)
spacing
unless skip(/\>/)
parse_error! "expected '>'"
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/counter/client.rb new/lib/fluent/counter/client.rb
--- old/lib/fluent/counter/client.rb 2019-01-07 04:07:42.000000000 +0100
+++ new/lib/fluent/counter/client.rb 2019-03-19 14:48:00.000000000 +0100
@@ -288,7 +288,7 @@
def join
until @set
@mutex.synchronize do
- @loop.run_once(0.0001) # retun a lock as soon as possible
+ @loop.run_once(0.0001) # return a lock as soon as possible
end
end
end
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/base.rb new/lib/fluent/plugin/base.rb
--- old/lib/fluent/plugin/base.rb 2019-01-07 04:07:42.000000000 +0100
+++ new/lib/fluent/plugin/base.rb 2019-03-19 14:48:00.000000000 +0100
@@ -52,7 +52,12 @@
def configure(conf)
if conf.respond_to?(:for_this_worker?) && conf.for_this_worker?
- system_config_override(workers: 1)
+ workers = if conf.target_worker_ids && !conf.target_worker_ids.empty?
+ conf.target_worker_ids.size
+ else
+ 1
+ end
+ system_config_override(workers: workers)
end
super
@_state ||= State.new(false, false, false, false, false, false, false, false, false)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/compressable.rb new/lib/fluent/plugin/compressable.rb
--- old/lib/fluent/plugin/compressable.rb 2019-01-07 04:07:42.000000000 +0100
+++ new/lib/fluent/plugin/compressable.rb 2019-03-19 14:48:00.000000000 +0100
@@ -44,7 +44,7 @@
# check compressed_data(String) is 0 length
compressed_data
when output_io
- # exeucte after checking compressed_data is empty or not
+ # execute after checking compressed_data is empty or not
io = StringIO.new(compressed_data)
io_decompress(io, output_io)
else
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/in_http.rb new/lib/fluent/plugin/in_http.rb
--- old/lib/fluent/plugin/in_http.rb 2019-01-07 04:07:42.000000000 +0100
+++ new/lib/fluent/plugin/in_http.rb 2019-03-19 14:48:00.000000000 +0100
@@ -163,6 +163,7 @@
# Skip nil record
if record.nil?
+ log.debug { "incoming event is invalid: path=#{path_info} params=#{params.to_json}" }
if @respond_with_empty_img
return ["200 OK", {'Content-Type'=>'image/gif; charset=utf-8'}, EMPTY_GIF_IMAGE]
else
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/in_syslog.rb new/lib/fluent/plugin/in_syslog.rb
--- old/lib/fluent/plugin/in_syslog.rb 2019-01-07 04:07:42.000000000 +0100
+++ new/lib/fluent/plugin/in_syslog.rb 2019-03-19 14:48:00.000000000 +0100
@@ -169,6 +169,10 @@
num = Integer(buffer[pos..idx])
pos = idx + num
msg = buffer[idx + 1...pos]
+ if msg.size < num - 1
+ pos = pos - num - num.to_s.size
+ break
+ end
message_handler(msg, conn)
end
else
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/in_tail.rb new/lib/fluent/plugin/in_tail.rb
--- old/lib/fluent/plugin/in_tail.rb 2019-01-07 04:07:42.000000000 +0100
+++ new/lib/fluent/plugin/in_tail.rb 2019-03-19 14:48:00.000000000 +0100
@@ -171,6 +171,9 @@
@encoding = parse_encoding_param(@encoding) if @encoding
@from_encoding = parse_encoding_param(@from_encoding) if @from_encoding
+ if @encoding == @from_encoding
+ log.warn "'encoding' and 'from_encoding' are same encoding. No effect"
+ end
end
def parse_encoding_param(encoding_name)
@@ -657,6 +660,7 @@
def initialize(from_encoding, encoding)
@from_encoding = from_encoding
@encoding = encoding
+ @need_enc = from_encoding != encoding
@buffer = ''.force_encoding(from_encoding)
@eol = "\n".encode(from_encoding).freeze
end
@@ -682,11 +686,13 @@
end
def convert(s)
- if @from_encoding == @encoding
- s
+ if @need_enc
+ s.encode!(@encoding, @from_encoding)
else
- s.encode(@encoding, @from_encoding)
+ s
end
+ rescue
+ s.encode!(@encoding, @from_encoding, :invalid => :replace, :undef => :replace)
end
def next_line
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/in_tcp.rb new/lib/fluent/plugin/in_tcp.rb
--- old/lib/fluent/plugin/in_tcp.rb 2019-01-07 04:07:42.000000000 +0100
+++ new/lib/fluent/plugin/in_tcp.rb 2019-03-19 14:48:00.000000000 +0100
@@ -41,11 +41,15 @@
def configure(conf)
compat_parameters_convert(conf, :parser)
+ parser_config = conf.elements('parse').first
+ unless parser_config
+ raise Fluent::ConfigError, "<parse> section is required."
+ end
super
@_event_loop_blocking_timeout = @blocking_timeout
@source_hostname_key ||= @source_host_key if @source_host_key
- @parser = parser_create
+ @parser = parser_create(conf: parser_config)
end
def multi_workers_ready?
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/in_udp.rb new/lib/fluent/plugin/in_udp.rb
--- old/lib/fluent/plugin/in_udp.rb 2019-01-07 04:07:42.000000000 +0100
+++ new/lib/fluent/plugin/in_udp.rb 2019-03-19 14:48:00.000000000 +0100
@@ -47,12 +47,16 @@
def configure(conf)
compat_parameters_convert(conf, :parser)
+ parser_config = conf.elements('parse').first
+ unless parser_config
+ raise Fluent::ConfigError, "<parse> section is required."
+ end
super
@_event_loop_blocking_timeout = @blocking_timeout
@source_hostname_key ||= @source_host_key if @source_host_key
@message_length_limit = @body_size_limit if @body_size_limit
- @parser = parser_create
+ @parser = parser_create(conf: parser_config)
end
def multi_workers_ready?
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/out_file.rb new/lib/fluent/plugin/out_file.rb
--- old/lib/fluent/plugin/out_file.rb 2019-01-07 04:07:42.000000000 +0100
+++ new/lib/fluent/plugin/out_file.rb 2019-03-19 14:48:00.000000000 +0100
@@ -71,6 +71,10 @@
attr_accessor :last_written_path # for tests
module SymlinkBufferMixin
+ def output_plugin_for_symlink=(output_plugin)
+ @_output_plugin_for_symlink = output_plugin
+ end
+
def symlink_path=(path)
@_symlink_path = path
end
@@ -83,7 +87,7 @@
# These chunks will be enqueued immediately, and will be flushed soon.
latest_metadata = metadata_list.select{|m| m.timekey }.sort_by(&:timekey).last
if chunk.metadata == latest_metadata
- FileUtils.ln_sf(chunk.path, @_symlink_path)
+ FileUtils.ln_sf(chunk.path, @_output_plugin_for_symlink.extract_placeholders(@_symlink_path, chunk))
end
chunk
end
@@ -161,6 +165,7 @@
else
@buffer.extend SymlinkBufferMixin
@buffer.symlink_path = @symlink_path
+ @buffer.output_plugin_for_symlink = self
end
end
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/out_forward.rb new/lib/fluent/plugin/out_forward.rb
--- old/lib/fluent/plugin/out_forward.rb 2019-01-07 04:07:42.000000000 +0100
+++ new/lib/fluent/plugin/out_forward.rb 2019-03-19 14:48:00.000000000 +0100
@@ -102,7 +102,7 @@
desc 'The client private key path for TLS.'
config_param :tls_client_private_key_path, :string, default: nil
desc 'The client private key passphrase for TLS.'
- config_param :tls_client_private_key_passphrase, :string, default: nil
+ config_param :tls_client_private_key_passphrase, :string, default: nil, secret: true
config_section :security, required: false, multi: false do
desc 'The hostname'
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/output.rb new/lib/fluent/plugin/output.rb
--- old/lib/fluent/plugin/output.rb 2019-01-07 04:07:42.000000000 +0100
+++ new/lib/fluent/plugin/output.rb 2019-03-19 14:48:00.000000000 +0100
@@ -364,7 +364,9 @@
raise Fluent::ConfigError, "Invalid <secondary> section for non-buffered plugin" unless @buffering
raise Fluent::ConfigError, "<secondary> section cannot have <buffer> section" if @secondary_config.buffer
raise Fluent::ConfigError, "<secondary> section cannot have <secondary> section" if @secondary_config.secondary
- raise Fluent::ConfigError, "<secondary> section and 'retry_forever' are exclusive" if @buffer_config.retry_forever
+ if @buffer_config.retry_forever
+ log.warn "<secondary> with 'retry_forever', only unrecoverable errors are moved to secondary"
+ end
secondary_type = @secondary_config[:@type]
unless secondary_type
@@ -1081,7 +1083,7 @@
end
end
- UNRECOVERABLE_ERRORS = [Fluent::UnrecoverableError, TypeError, ArgumentError, NoMethodError]
+ UNRECOVERABLE_ERRORS = [Fluent::UnrecoverableError, TypeError, ArgumentError, NoMethodError, MessagePack::UnpackError]
def try_flush
chunk = @buffer.dequeue_chunk
@@ -1334,7 +1336,7 @@
end
if @chunk_key_time
if !value_for_interval || @buffer_config.timekey < value_for_interval
- value_for_interval = @buffer_config.timekey
+ value_for_interval = [@buffer_config.timekey, @buffer_config.timekey_wait].min
end
end
unless value_for_interval
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/parser_regexp.rb new/lib/fluent/plugin/parser_regexp.rb
--- old/lib/fluent/plugin/parser_regexp.rb 2019-01-07 04:07:42.000000000 +0100
+++ new/lib/fluent/plugin/parser_regexp.rb 2019-03-19 14:48:00.000000000 +0100
@@ -40,6 +40,10 @@
@expression = Regexp.compile(@expression.source, options)
end
@regexp = @expression # For backward compatibility
+
+ if @expression.named_captures.empty?
+ raise Fluent::ConfigError, "No named captures in 'expression' parameter. The regexp must have at least one named capture"
+ end
end
def parse(text)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin_helper/retry_state.rb new/lib/fluent/plugin_helper/retry_state.rb
--- old/lib/fluent/plugin_helper/retry_state.rb 2019-01-07 04:07:42.000000000 +0100
+++ new/lib/fluent/plugin_helper/retry_state.rb 2019-03-19 14:48:00.000000000 +0100
@@ -53,10 +53,6 @@
@randomize = randomize
@randomize_width = randomize_width
- if forever && secondary
- raise "BUG: forever and secondary are exclusive to each other"
- end
-
@forever = forever
@max_steps = max_steps
@@ -118,12 +114,12 @@
end
def secondary?
- @secondary && (@current == :secondary || current_time >= @secondary_transition_at)
+ !@forever && @secondary && (@current == :secondary || current_time >= @secondary_transition_at)
end
def step
@steps += 1
- if @secondary && @current != :secondary && current_time >= @secondary_transition_at
+ if !@forever && @secondary && @current != :secondary && current_time >= @secondary_transition_at
@current = :secondary
@secondary_transition_steps = @steps
end
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin_helper/server.rb new/lib/fluent/plugin_helper/server.rb
--- old/lib/fluent/plugin_helper/server.rb 2019-01-07 04:07:42.000000000 +0100
+++ new/lib/fluent/plugin_helper/server.rb 2019-03-19 14:48:00.000000000 +0100
@@ -523,7 +523,7 @@
def on_readable_without_sock
begin
data = @sock.recv(@max_bytes, @flags)
- rescue Errno::EAGAIN, Errno::EWOULDBLOCK, Errno::EINTR, Errno::ECONNRESET
+ rescue Errno::EAGAIN, Errno::EWOULDBLOCK, Errno::EINTR, Errno::ECONNRESET, IOError, Errno::EBADF
return
end
@callback.call(data)
@@ -536,7 +536,7 @@
def on_readable_with_sock
begin
data, addr = @sock.recvfrom(@max_bytes)
- rescue Errno::EAGAIN, Errno::EWOULDBLOCK, Errno::EINTR, Errno::ECONNRESET
+ rescue Errno::EAGAIN, Errno::EWOULDBLOCK, Errno::EINTR, Errno::ECONNRESET, IOError, Errno::EBADF
return
end
@callback.call(data, UDPCallbackSocket.new(@sock, addr, close_socket: @close_socket))
@@ -722,7 +722,7 @@
return true
end
- rescue Errno::ECONNRESET, OpenSSL::SSL::SSLError => e
+ rescue Errno::EPIPE, Errno::ECONNRESET, OpenSSL::SSL::SSLError => e
@log.trace "unexpected error before accepting TLS connection", error: e
close rescue nil
end
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/root_agent.rb new/lib/fluent/root_agent.rb
--- old/lib/fluent/root_agent.rb 2019-01-07 04:07:42.000000000 +0100
+++ new/lib/fluent/root_agent.rb 2019-03-19 14:48:00.000000000 +0100
@@ -64,26 +64,64 @@
attr_reader :labels
def configure(conf)
+ used_worker_ids = []
+ available_worker_ids = (0..Fluent::Engine.system_config.workers - 1).to_a
# initialize <worker> elements
conf.elements(name: 'worker').each do |e|
target_worker_id_str = e.arg
if target_worker_id_str.empty?
- raise ConfigError, "Missing worker id on <worker> directive"
+ raise Fluent::ConfigError, "Missing worker id on <worker> directive"
end
- target_worker_id = target_worker_id_str.to_i
- if target_worker_id < 0 || target_worker_id > (Fluent::Engine.system_config.workers - 1)
- raise ConfigError, "worker id #{target_worker_id} specified by <worker> directive is not allowed. Available worker id is between 0 and #{(Fluent::Engine.system_config.workers - 1)}"
- end
+ target_worker_ids = target_worker_id_str.split("-")
+ if target_worker_ids.size == 2
+ first_worker_id = target_worker_ids.first.to_i
+ last_worker_id = target_worker_ids.last.to_i
+ if first_worker_id > last_worker_id
+ raise Fluent::ConfigError, "greater first_worker_id<#{first_worker_id}> than last_worker_id<#{last_worker_id}> specified by <worker> directive is not allowed. Available multi worker assign syntax is -"
+ end
+ target_worker_ids = []
+ first_worker_id.step(last_worker_id, 1) do |worker_id|
+ target_worker_id = worker_id.to_i
+ target_worker_ids << target_worker_id
+
+ if target_worker_id < 0 || target_worker_id > (Fluent::Engine.system_config.workers - 1)
+ raise Fluent::ConfigError, "worker id #{target_worker_id} specified by <worker> directive is not allowed. Available worker id is between 0 and #{(Fluent::Engine.system_config.workers - 1)}"
+ end
+ available_worker_ids.delete(target_worker_id) if available_worker_ids.include?(target_worker_id)
+ if used_worker_ids.include?(target_worker_id) && !Fluent::Engine.dry_run_mode
+ raise Fluent::ConfigError, "specified worker_id<#{worker_id}> collisions is detected on <worker> directive. Available worker id(s): #{available_worker_ids}"
+ end
+ used_worker_ids << target_worker_id
+
+ e.elements.each do |elem|
+ unless ['source', 'match', 'filter', 'label'].include?(elem.name)
+ raise Fluent::ConfigError, "<worker> section cannot have <#{elem.name}> directive"
+ end
+ end
+
+ # On dry_run mode, all worker sections have to be configured on supervisor (recognized as worker_id = 0).
+ target_worker_ids = [0] if Fluent::Engine.dry_run_mode
+
+ unless target_worker_ids.empty?
+ e.set_target_worker_ids(target_worker_ids.uniq)
+ end
+ end
+ else
+ target_worker_id = target_worker_id_str.to_i
+ if target_worker_id < 0 || target_worker_id > (Fluent::Engine.system_config.workers - 1)
+ raise Fluent::ConfigError, "worker id #{target_worker_id} specified by <worker> directive is not allowed. Available worker id is between 0 and #{(Fluent::Engine.system_config.workers - 1)}"
+ end
- ## On dry_run mode, all worker sections have to be configured on supervisor (recognized as worker_id = 0).
- target_worker_id = 0 if Fluent::Engine.dry_run_mode
+ ## On dry_run mode, all worker sections have to be configured on supervisor (recognized as worker_id = 0).
+ target_worker_id = 0 if Fluent::Engine.dry_run_mode
- e.elements.each do |elem|
- unless ['source', 'match', 'filter', 'label'].include?(elem.name)
- raise ConfigError, "<worker> section cannot have <#{elem.name}> directive"
+ e.elements.each do |elem|
+ unless ['source', 'match', 'filter', 'label'].include?(elem.name)
+ raise Fluent::ConfigError, "<worker> section cannot have <#{elem.name}> directive"
+ end
+ elem.set_target_worker_id(target_worker_id)
end
- elem.set_target_worker_id(target_worker_id)
end
conf += e
end
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/supervisor.rb new/lib/fluent/supervisor.rb
--- old/lib/fluent/supervisor.rb 2019-01-07 04:07:42.000000000 +0100
+++ new/lib/fluent/supervisor.rb 2019-03-19 14:48:00.000000000 +0100
@@ -716,7 +716,13 @@
end
def main_process(&block)
- Process.setproctitle("worker:#{@process_name}") if @process_name
+ if @process_name
+ if @workers > 1
+ Process.setproctitle("worker:#{@process_name}#{ENV['SERVERENGINE_WORKER_ID']}")
+ else
+ Process.setproctitle("worker:#{@process_name}")
+ end
+ end
unrecoverable_error = false
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/version.rb new/lib/fluent/version.rb
--- old/lib/fluent/version.rb 2019-01-07 04:07:42.000000000 +0100
+++ new/lib/fluent/version.rb 2019-03-19 14:48:00.000000000 +0100
@@ -16,6 +16,6 @@
module Fluent
- VERSION = '1.3.3'
+ VERSION = '1.4.1'
end
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/metadata new/metadata
--- old/metadata 2019-01-07 04:07:42.000000000 +0100
+++ new/metadata 2019-03-19 14:48:00.000000000 +0100
@@ -1,14 +1,14 @@
--- !ruby/object:Gem::Specification
name: fluentd
version: !ruby/object:Gem::Version
- version: 1.3.3
+ version: 1.4.1
platform: ruby
authors:
- Sadayuki Furuhashi
autorequire:
bindir: bin
cert_chain: []
-date: 2019-01-07 00:00:00.000000000 Z
+date: 2019-03-19 00:00:00.000000000 Z
dependencies:
- !ruby/object:Gem::Dependency
name: msgpack
@@ -329,6 +329,7 @@
extra_rdoc_files: []
files:
- ".github/ISSUE_TEMPLATE.md"
+- ".github/PULL_REQUEST_TEMPLATE.md"
- ".gitignore"
- ".travis.yml"
- ADOPTERS.md
@@ -337,6 +338,7 @@
- CONTRIBUTING.md
- GOVERNANCE.md
- Gemfile
+- GithubWorkflow.md
- LICENSE
- MAINTAINERS.md
- README.md
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/command/test_fluentd.rb new/test/command/test_fluentd.rb
--- old/test/command/test_fluentd.rb 2019-01-07 04:07:42.000000000 +0100
+++ new/test/command/test_fluentd.rb 2019-03-19 14:48:00.000000000 +0100
@@ -340,7 +340,7 @@
end
end
- sub_test_case 'configured to suppress configration dump' do
+ sub_test_case 'configured to suppress configuration dump' do
setup do
@basic_conf = < "dummy" })
regexp = parser.instance_variable_get("@regexp")
assert_equal(options, regexp.options)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/config/test_config_parser.rb new/test/config/test_config_parser.rb
--- old/test/config/test_config_parser.rb 2019-01-07 04:07:42.000000000 +0100
+++ new/test/config/test_config_parser.rb 2019-03-19 14:48:00.000000000 +0100
@@ -329,6 +329,36 @@
end
end
+ sub_test_case "Embedded Ruby Code in section attributes" do
+ setup do
+ ENV["EMBEDDED_VAR"] = "embedded"
+ ENV["NESTED_EMBEDDED_VAR"] = "nested-embedded"
+ @hostname = Socket.gethostname
+ end
+
+ teardown do
+ ENV["EMBEDDED_VAR"] = nil
+ ENV["NESTED_EMBEDDED_VAR"] = nil
+ end
+
+ test "embedded Ruby code should be expanded" do
+ assert_text_parsed_as(root(
+ e("test", 'embedded', {'key'=>'1'}, [
+ e('nested1', 'nested-embedded'),
+ e('nested2', "#{@hostname}")
+ ])), <<-EOF
+
+ key 1
+
+ </nested1>
+
+ </nested2>
+ </test>
+ EOF
+ )
+ end
+ end
+
# port from test_config.rb
sub_test_case '@include parsing' do
TMP_DIR = File.dirname(__FILE__) + "/tmp/v1_config#{ENV['TEST_ENV_NUMBER']}"
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/config/test_configurable.rb new/test/config/test_configurable.rb
--- old/test/config/test_configurable.rb 2019-01-07 04:07:42.000000000 +0100
+++ new/test/config/test_configurable.rb 2019-03-19 14:48:00.000000000 +0100
@@ -733,7 +733,7 @@
assert_nothing_raised { b4.configure(config_element('ROOT', '', BASE_ATTRS, [d1.dup, d2.dup, d3.dup, d3.dup])) }
end
- test 'constructs confuguration object tree for Base3' do
+ test 'constructs configuration object tree for Base3' do
conf = config_element(
'ROOT',
'',
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/config/test_element.rb new/test/config/test_element.rb
--- old/test/config/test_element.rb 2019-01-07 04:07:42.000000000 +0100
+++ new/test/config/test_element.rb 2019-03-19 14:48:00.000000000 +0100
@@ -157,7 +157,7 @@
Fluent::Config::Element.new('ROOT', 'mydata', {}, [])],
"differ keys" => [Fluent::Config::Element.new('ROOT', 'mydata', {}, []),
Fluent::Config::Element.new('ROOT', 'mydata', {"k1" => "v1"}, [])],
- "differ elemnts" =>
+ "differ elements" =>
[Fluent::Config::Element.new('ROOT', 'mydata', {"k1" => "v1"}, []),
Fluent::Config::Element.new('ROOT', 'mydata', {"k1" => "v1"}, [
Fluent::Config::Element.new('test', 'mydata', {'k3' => 'v3'}, [])
@@ -406,11 +406,11 @@
test 'set target_worker_id recursively' do
e = element('label', '@mytest', {}, [ element('filter', '**'), element('match', '**', {}, [ element('store'), element('store') ]) ])
e.set_target_worker_id(1)
- assert_equal 1, e.target_worker_id
- assert_equal 1, e.elements[0].target_worker_id
- assert_equal 1, e.elements[1].target_worker_id
- assert_equal 1, e.elements[1].elements[0].target_worker_id
- assert_equal 1, e.elements[1].elements[1].target_worker_id
+ assert_equal [1], e.target_worker_ids
+ assert_equal [1], e.elements[0].target_worker_ids
+ assert_equal [1], e.elements[1].target_worker_ids
+ assert_equal [1], e.elements[1].elements[0].target_worker_ids
+ assert_equal [1], e.elements[1].elements[1].target_worker_ids
end
end
@@ -434,12 +434,24 @@
assert e.for_this_worker?
end
+ test 'target_worker_ids includes current worker_id' do
+ e = element()
+ e.set_target_worker_ids([0])
+ assert e.for_this_worker?
+ end
+
test 'target_worker_id != current worker_id' do
e = element()
e.set_target_worker_id(1)
assert_false e.for_this_worker?
end
+ test 'target_worker_ids does not includes current worker_id' do
+ e = element()
+ e.set_target_worker_ids([1, 2])
+ assert_false e.for_this_worker?
+ end
+
test "doesn't have target_worker_id" do
e = element()
assert_false e.for_this_worker?
@@ -453,12 +465,24 @@
assert_false e.for_another_worker?
end
+ test 'target_worker_ids contains current worker_id' do
+ e = element()
+ e.set_target_worker_ids([0, 1])
+ assert_false e.for_another_worker?
+ end
+
test 'target_worker_id != current worker_id' do
e = element()
e.set_target_worker_id(1)
assert e.for_another_worker?
end
+ test 'target_worker_ids does not contains current worker_id' do
+ e = element()
+ e.set_target_worker_ids([1, 2])
+ assert e.for_another_worker?
+ end
+
test "doesn't have target_worker_id" do
e = element()
assert_false e.for_another_worker?
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/counter/test_store.rb new/test/counter/test_store.rb
--- old/test/counter/test_store.rb 2019-01-07 04:07:42.000000000 +0100
+++ new/test/counter/test_store.rb 2019-03-19 14:48:00.000000000 +0100
@@ -89,7 +89,7 @@
assert_equal nil, @store.get('unknown_key')
end
- test "raise a error when when a passed key doesn't exist and raise_error option is true" do
+ test "raise a error when a passed key doesn't exist and raise_error option is true" do
assert_raise Fluent::Counter::UnknownKey do
@store.get('unknown_key', raise_error: true)
end
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_buffer.rb new/test/plugin/test_buffer.rb
--- old/test/plugin/test_buffer.rb 2019-01-07 04:07:42.000000000 +0100
+++ new/test/plugin/test_buffer.rb 2019-03-19 14:48:00.000000000 +0100
@@ -211,7 +211,7 @@
assert_equal 2, @p.queued_num[@dm1]
end
- test '#close closes all chunks in in dequeued, enqueued and staged' do
+ test '#close closes all chunks in dequeued, enqueued and staged' do
dmx = create_metadata(Time.parse('2016-04-11 15:50:00 +0000').to_i, nil, nil)
cx = create_chunk(dmx, ["x" * 1024])
@p.dequeued[cx.unique_id] = cx
@@ -1027,7 +1027,7 @@
##### 900 + 9500 + 9900 * 4 == 5000 + 45000
end
- test '#write raises BufferChunkOverflowError if a record is biggar than chunk limit size' do
+ test '#write raises BufferChunkOverflowError if a record is bigger than chunk limit size' do
assert_equal [@dm0], @p.stage.keys
assert_equal [], @p.queue.map(&:metadata)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_filter.rb new/test/plugin/test_filter.rb
--- old/test/plugin/test_filter.rb 2019-01-07 04:07:43.000000000 +0100
+++ new/test/plugin/test_filter.rb 2019-03-19 14:48:00.000000000 +0100
@@ -47,7 +47,7 @@
end
end
class InvalidPlugin < Fluent::Plugin::Filter
- # Because of implemnting `filter_with_time` and `filter` methods
+ # Because of implementing `filter_with_time` and `filter` methods
def filter_with_time(tag, time, record); end
def filter(tag, time, record); end
end
@@ -277,7 +277,7 @@
end
end
- sub_test_case 'filter plugins that is implmented `filter_with_time`' do
+ sub_test_case 'filter plugins that is implemented `filter_with_time`' do
setup do
Fluent::Test.setup
@p = FluentPluginFilterTest::NumDoublePluginWithTime.new
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_filter_record_transformer.rb new/test/plugin/test_filter_record_transformer.rb
--- old/test/plugin/test_filter_record_transformer.rb 2019-01-07 04:07:43.000000000 +0100
+++ new/test/plugin/test_filter_record_transformer.rb 2019-03-19 14:48:00.000000000 +0100
@@ -229,7 +229,7 @@
end
%w[yes no].each do |enable_ruby|
- test "hostname with enble_ruby #{enable_ruby}" do
+ test "hostname with enable_ruby #{enable_ruby}" do
config = %[
enable_ruby #{enable_ruby}
<record>
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_in_tail.rb new/test/plugin/test_in_tail.rb
--- old/test/plugin/test_in_tail.rb 2019-01-07 04:07:43.000000000 +0100
+++ new/test/plugin/test_in_tail.rb 2019-03-19 14:48:00.000000000 +0100
@@ -649,6 +649,26 @@
assert_equal(Encoding::UTF_8, events[0][2]['message'].encoding)
end
+ def test_encoding_with_bad_character
+ conf = config_element(
+ "", "", {
+ "format" => "/(?<message>.*)/",
+ "read_from_head" => "true",
+ "from_encoding" => "ASCII-8BIT",
+ "encoding" => "utf-8"
+ })
+ d = create_driver(conf)
+
+ d.run(expect_emits: 1) do
+ File.open("#{TMP_DIR}/tail.txt", "w") { |f|
+ f.write "te\x86st\n"
+ }
+ end
+
+ events = d.events
+ assert_equal("te\uFFFDst", events[0][2]['message'])
+ assert_equal(Encoding::UTF_8, events[0][2]['message'].encoding)
+ end
sub_test_case "multiline" do
data(flat: MULTILINE_CONFIG,
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_in_tcp.rb new/test/plugin/test_in_tcp.rb
--- old/test/plugin/test_in_tcp.rb 2019-01-07 04:07:43.000000000 +0100
+++ new/test/plugin/test_in_tcp.rb 2019-03-19 14:48:00.000000000 +0100
@@ -48,6 +48,12 @@
assert_equal "\n", d.instance.delimiter
end
+ test ' configure w/o parse section' do
+ assert_raise(Fluent::ConfigError.new("<parse> section is required.")) {
+ create_driver(BASE_CONFIG)
+ }
+ end
+
test_case_data = {
'none' => {
'format' => 'none',
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_in_udp.rb new/test/plugin/test_in_udp.rb
--- old/test/plugin/test_in_udp.rb 2019-01-07 04:07:43.000000000 +0100
+++ new/test/plugin/test_in_udp.rb 2019-03-19 14:48:00.000000000 +0100
@@ -59,6 +59,12 @@
assert_equal nil, d.instance.receive_buffer_size
end
+ test ' configure w/o parse section' do
+ assert_raise(Fluent::ConfigError.new("<parse> section is required.")) {
+ create_driver(BASE_CONFIG)
+ }
+ end
+
data(
'ipv4' => [CONFIG, '127.0.0.1', :ipv4],
'ipv6' => [IPv6_CONFIG, '::1', :ipv6],
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_out_file.rb new/test/plugin/test_out_file.rb
--- old/test/plugin/test_out_file.rb 2019-01-07 04:07:43.000000000 +0100
+++ new/test/plugin/test_out_file.rb 2019-03-19 14:48:00.000000000 +0100
@@ -14,7 +14,6 @@
end
TMP_DIR = File.expand_path(File.dirname(__FILE__) + "/../tmp/out_file#{ENV['TEST_ENV_NUMBER']}")
- SYMLINK_PATH = File.expand_path("#{TMP_DIR}/current")
CONFIG = %[
path #{TMP_DIR}/out_file_test
@@ -663,15 +662,43 @@
end
end
- test 'symlink' do
- omit "Windows doesn't support symlink" if Fluent.windows?
- conf = CONFIG + %[
- symlink_path #{SYMLINK_PATH}
- ]
- symlink_path = "#{SYMLINK_PATH}"
+ SYMLINK_PATH = File.expand_path("#{TMP_DIR}/current")
+
+ sub_test_case 'symlink' do
+ test 'static symlink' do
+ omit "Windows doesn't support symlink" if Fluent.windows?
+ conf = CONFIG + %[
+ symlink_path #{SYMLINK_PATH}
+ ]
+ symlink_path = "#{SYMLINK_PATH}"
+
+ d = create_driver(conf)
+ begin
+ run_and_check(d, symlink_path)
+ ensure
+ FileUtils.rm_rf(symlink_path)
+ end
+ end
+
+ test 'symlink with placeholders' do
+ omit "Windows doesn't support symlink" if Fluent.windows?
+ conf = %[
+ path #{TMP_DIR}/${tag}/out_file_test
+ symlink_path #{SYMLINK_PATH}-${tag}
+
+ </buffer>
+ ]
+ symlink_path = "#{SYMLINK_PATH}-tag"
+
+ d = create_driver(conf)
+ begin
+ run_and_check(d, symlink_path)
+ ensure
+ FileUtils.rm_rf(symlink_path)
+ end
+ end
- d = create_driver(conf)
- begin
+ def run_and_check(d, symlink_path)
d.run(default_tag: 'tag') do
es = Fluent::OneEventStream.new(event_time("2011-01-02 13:14:15 UTC"), {"a"=>1})
d.feed(es)
@@ -688,8 +715,6 @@
meta = d.instance.metadata('tag', event_time("2011-01-03 14:15:16 UTC"), {})
assert_equal d.instance.buffer.instance_eval{ @stage[meta].path }, File.readlink(symlink_path)
end
- ensure
- FileUtils.rm_rf(symlink_path)
end
end
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_out_secondary_file.rb new/test/plugin/test_out_secondary_file.rb
--- old/test/plugin/test_out_secondary_file.rb 2019-01-07 04:07:43.000000000 +0100
+++ new/test/plugin/test_out_secondary_file.rb 2019-03-19 14:48:00.000000000 +0100
@@ -39,7 +39,7 @@
c.configure(conf)
end
- sub_test_case 'configture' do
+ sub_test_case 'configure' do
test 'default configuration' do
d = create_driver %[directory #{TMP_DIR}]
assert_equal 'dump.bin', d.instance.basename
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_output.rb new/test/plugin/test_output.rb
--- old/test/plugin/test_output.rb 2019-01-07 04:07:43.000000000 +0100
+++ new/test/plugin/test_output.rb 2019-03-19 14:48:00.000000000 +0100
@@ -547,7 +547,7 @@
assert_equal 86400, s
assert_equal :day, t
assert_equal '%d', e
- s, t, e = @i.get_placeholders_time("my birthiday! at %F")
+ s, t, e = @i.get_placeholders_time("my birthday! at %F")
assert_equal 86400, s
assert_equal :day, t
assert_equal '%d', e
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_output_as_buffered.rb new/test/plugin/test_output_as_buffered.rb
--- old/test/plugin/test_output_as_buffered.rb 2019-01-07 04:07:43.000000000 +0100
+++ new/test/plugin/test_output_as_buffered.rb 2019-03-19 14:48:00.000000000 +0100
@@ -1074,6 +1074,44 @@
end
end
+ sub_test_case 'buffered output with large timekey and small timekey_wait' do
+ setup do
+ chunk_key = 'time'
+ hash = {
+ 'timekey' => 86400, # per 1 day
+ 'timekey_wait' => 10, # 10 seconds delay for flush
+ 'flush_thread_count' => 1,
+ 'flush_thread_burst_interval' => 0.01,
+ }
+ @i = create_output(:buffered)
+ @i.configure(config_element('ROOT','',{},[config_element('buffer',chunk_key,hash)]))
+ @i.start
+ @i.after_start
+ end
+
+ test '#configure raises config error if timekey is not specified' do
+ Timecop.freeze( Time.parse('2019-02-08 00:01:00 +0900') )
+ ary = []
+ @i.register(:write){|chunk| ary << chunk.read }
+ @i.thread_wait_until_start
+ events = [
+ [event_time('2019-02-08 00:02:00 +0900'), {"message" => "foobar"}]
+ ]
+ @i.emit_events("test.tag", Fluent::ArrayEventStream.new(events))
+ @i.enqueue_thread_wait
+ assert{ @i.write_count == 0 }
+
+ Timecop.freeze( Time.parse('2019-02-09 00:00:08 +0900') )
+ @i.enqueue_thread_wait
+ assert{ @i.write_count == 0 }
+
+ Timecop.freeze( Time.parse('2019-02-09 00:00:12 +0900') )
+ # wirte should be called in few seconds since
+ # running interval of enque thread is timekey_wait / 11.0.
+ waiting(5){ sleep 0.1 until @i.write_count == 1 }
+ end
+ end
+
sub_test_case 'buffered output feature with tag key' do
setup do
chunk_key = 'tag'
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_output_as_buffered_backup.rb new/test/plugin/test_output_as_buffered_backup.rb
--- old/test/plugin/test_output_as_buffered_backup.rb 2019-01-07 04:07:43.000000000 +0100
+++ new/test/plugin/test_output_as_buffered_backup.rb 2019-03-19 14:48:00.000000000 +0100
@@ -177,7 +177,12 @@
}
end
- test 'backup chunk without secondary' do
+ data('unrecoverable error' => Fluent::UnrecoverableError,
+ 'type error' => TypeError,
+ 'argument error' => ArgumentError,
+ 'no method error' => NoMethodError,
+ 'msgpack unpack error' => MessagePack::UnpackError)
+ test 'backup chunk without secondary' do |error_class|
Fluent::SystemConfig.overwrite_system_config('root_dir' => TMP_DIR) do
id = 'backup_test'
hash = {
@@ -188,7 +193,7 @@
@i.configure(config_element('ROOT', '', {'@id' => id}, [config_element('buffer', 'tag', hash)]))
@i.register(:write) { |chunk|
chunk_id = chunk.unique_id;
- raise Fluent::UnrecoverableError, "yay, your #write must fail"
+ raise error_class, "yay, your #write must fail"
}
flush_chunks
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_output_as_buffered_retries.rb new/test/plugin/test_output_as_buffered_retries.rb
--- old/test/plugin/test_output_as_buffered_retries.rb 2019-01-07 04:07:43.000000000 +0100
+++ new/test/plugin/test_output_as_buffered_retries.rb 2019-03-19 14:48:00.000000000 +0100
@@ -706,16 +706,20 @@
end
sub_test_case 'buffered output configured as retry_forever' do
- test 'configuration error will be raised if secondary section is configured' do
+ setup do
+ Fluent::Plugin.register_output('output_retries_secondary_test', FluentPluginOutputAsBufferedRetryTest::DummyFullFeatureOutput2)
+ end
+
+ test 'warning logs are generated if secondary section is configured' do
chunk_key = 'tag'
hash = {
'retry_forever' => true,
'retry_randomize' => false,
}
i = create_output()
- assert_raise Fluent::ConfigError do
- i.configure(config_element('ROOT','',{},[config_element('buffer',chunk_key,hash),config_element('secondary','')]))
- end
+ i.configure(config_element('ROOT','',{},[config_element('buffer',chunk_key,hash),config_element('secondary','', {'@type' => 'output_retries_secondary_test'})]))
+ logs = i.log.out.logs
+ assert { logs.any? { |l| l.include?("<secondary> with 'retry_forever', only unrecoverable errors are moved to secondary") } }
end
test 'retry_timeout and retry_max_times will be ignored if retry_forever is true for exponential backoff' do
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_parser_regexp.rb new/test/plugin/test_parser_regexp.rb
--- old/test/plugin/test_parser_regexp.rb 2019-01-07 04:07:43.000000000 +0100
+++ new/test/plugin/test_parser_regexp.rb 2019-03-19 14:48:00.000000000 +0100
@@ -153,6 +153,15 @@
end
sub_test_case "configure" do
+ def test_bad_expression
+ conf = {
+ 'expression' => %q!/.*/!,
+ }
+ assert_raise Fluent::ConfigError do
+ create_driver(conf)
+ end
+ end
+
def test_default_options
conf = {
'expression' => %q!/^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] \[(?<date>[^\]]*)\] "(?<flag>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)$/!,
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_parser_syslog.rb new/test/plugin/test_parser_syslog.rb
--- old/test/plugin/test_parser_syslog.rb 2019-01-07 04:07:43.000000000 +0100
+++ new/test/plugin/test_parser_syslog.rb 2019-03-19 14:48:00.000000000 +0100
@@ -276,7 +276,7 @@
def test_auto_with_legacy_syslog_message
@parser.configure(
'time_format' => '%b %d %M:%S:%H',
- 'mseeage_format' => 'auto',
+ 'message_format' => 'auto',
)
text = 'Feb 28 00:00:12 192.168.0.1 fluentd[11111]: [error] Syslog test'
@parser.instance.parse(text) do |time, record|
@@ -290,7 +290,7 @@
@parser.configure(
'time_format' => '%b %d %M:%S:%H',
'with_priority' => true,
- 'mseeage_format' => 'auto',
+ 'message_format' => 'auto',
)
text = '<6>Feb 28 12:00:00 192.168.0.1 fluentd[11111]: [error] Syslog test'
@parser.instance.parse(text) do |time, record|
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin_helper/test_record_accessor.rb new/test/plugin_helper/test_record_accessor.rb
--- old/test/plugin_helper/test_record_accessor.rb 2019-01-07 04:07:43.000000000 +0100
+++ new/test/plugin_helper/test_record_accessor.rb 2019-03-19 14:48:00.000000000 +0100
@@ -48,7 +48,7 @@
data("missing ']'" => "$['key1'",
"missing array index with dot" => "$.hello[]",
- "missing array index with braket" => "$[]",
+ "missing array index with bracket" => "$[]",
"more chars" => "$.key1[0]foo",
"whitespace char included key in dot notation" => "$.key[0].ke y",
"empty keys with dot" => "$.",
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin_helper/test_server.rb new/test/plugin_helper/test_server.rb
--- old/test/plugin_helper/test_server.rb 2019-01-07 04:07:43.000000000 +0100
+++ new/test/plugin_helper/test_server.rb 2019-03-19 14:48:00.000000000 +0100
@@ -767,7 +767,7 @@
def write_cert_and_key(cert_path, cert, key_path, key, passphrase)
File.open(cert_path, "w"){|f| f.write(cert.to_pem) }
- # Write the secret key (raw or ecnrypted by AES256) in PEM format
+ # Write the secret key (raw or encrypted by AES256) in PEM format
key_str = passphrase ? key.export(OpenSSL::Cipher.new("AES-256-CBC"), passphrase) : key.export
File.open(key_path, "w"){|f| f.write(key_str) }
File.chmod(0600, cert_path, key_path)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/test_output.rb new/test/test_output.rb
--- old/test/test_output.rb 2019-01-07 04:07:43.000000000 +0100
+++ new/test/test_output.rb 2019-03-19 14:48:00.000000000 +0100
@@ -129,7 +129,7 @@
end
def test_secondary_with_no_warn_log
- # ObjectBufferedOutput doesn't implemnt `custom_filter`
+ # ObjectBufferedOutput doesn't implement `custom_filter`
d = Fluent::Test::BufferedOutputTestDriver.new(Fluent::ObjectBufferedOutput)
mock(d.instance.log).warn("secondary type should be same with primary one",
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/test_plugin.rb new/test/test_plugin.rb
--- old/test/test_plugin.rb 2019-01-07 04:07:43.000000000 +0100
+++ new/test/test_plugin.rb 2019-03-19 14:48:00.000000000 +0100
@@ -97,7 +97,7 @@
output1: ['plugin_test_dummy1', Dummy1Output, :new_output],
output2: ['plugin_test_dummy2', Dummy2Output, :new_output],
)
- test 'retruns plugin instances of registered plugin classes' do |(type, klass, m)|
+ test 'returns plugin instances of registered plugin classes' do |(type, klass, m)|
instance = Fluent::Plugin.__send__(m, type)
assert_kind_of klass, instance
end
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/test_root_agent.rb new/test/test_root_agent.rb
--- old/test/test_root_agent.rb 2019-01-07 04:07:43.000000000 +0100
+++ new/test/test_root_agent.rb 2019-03-19 14:48:00.000000000 +0100
@@ -686,6 +686,54 @@
end
end
+ test 'raises configuration error for too big worker id on multi workers syntax' do
+ errmsg = "worker id 4 specified by <worker> directive is not allowed. Available worker id is between 0 and 3"
+ assert_raise Fluent::ConfigError.new(errmsg) do
+ conf = <<-EOC
+
+</worker>
+EOC
+ configure_ra(conf)
+ end
+ end
+
+ test 'raises configuration error for worker id collisions on multi workers syntax' do
+ errmsg = "specified worker_id<2> collisions is detected on <worker> directive. Available worker id(s): [3]"
+ assert_raise Fluent::ConfigError.new(errmsg) do
+ conf = <<-EOC
+
+</worker>
+
+</worker>
+EOC
+ configure_ra(conf)
+ end
+ end
+
+ test 'raises configuration error for worker id collisions on multi workers syntax when multi avaliable worker_ids are left' do
+ errmsg = "specified worker_id<1> collisions is detected on <worker> directive. Available worker id(s): [2, 3]"
+ assert_raise Fluent::ConfigError.new(errmsg) do
+ conf = <<-EOC
+
+</worker>
+
+</worker>
+EOC
+ configure_ra(conf)
+ end
+ end
+
+ test 'raises configuration error for too big worker id on invalid reversed multi workers syntax' do
+ errmsg = "greater first_worker_id<3> than last_worker_id<0> specified by <worker> directive is not allowed. Available multi worker assign syntax is -"
+ assert_raise Fluent::ConfigError.new(errmsg) do
+ conf = <<-EOC
+
+</worker>
+EOC
+ configure_ra(conf)
+ end
+ end
+
test 'raises configuration error for invalid elements as a child of worker section' do
errmsg = '<worker> section cannot have <system> directive'
assert_raise Fluent::ConfigError.new(errmsg) do
@@ -844,5 +892,33 @@
assert_equal 0, ra.labels.size
refute ra.error_collector
end
+
+ test 'with plugins for workers syntax should match worker_id equals to 2' do
+ conf = <<-EOC
+
+ <source>
+ @type forward
+ </source>
+
+ @type test_filter
+ @id test_filter
+ </filter>
+ <match pattern>
+ @type stdout
+ </match>
+