19

我想将我的用户直接发送到特定的日志组和过滤器，但我需要能够生成正确的 URL 格式。例如，这个网址

https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/
%252Fmy%252Flog%252Fgroup%252Fgoes%252Fhere/log-events/$3FfilterPattern$3D$255Bincoming_ip$252C$2Buser_name$252C$2Buser_ip$2B$252C$2Btimestamp$252C$2Brequest$2B$2521$253D$2B$2522GET$2B$252Fhealth_checks$252Fall$2B*$2522$252C$2Bstatus_code$2B$253D$2B5*$2B$257C$257C$2Bstatus_code$2B$253D$2B429$252C$2Bbytes$252C$2Burl$252C$2Buser_agent$255D$26start$3D-172800000

将带您进入一个名为的日志组/my/log/group/goes/here，并在过去 2 天内使用此模式过滤消息：

[incoming_ip, user_name, user_ip , timestamp, request != "GET /health_checks/all *", status_code = 5* || status_code = 429, bytes, url, user_agent]

我可以对 URL 的一部分进行解码，但我不知道其他一些字符应该是什么（见下文），但这在我看来并不像任何标准的 HTML 编码。有谁知道这种 URL 格式的编码器/解码器？

%252F == /
$252C == ,
$255B == [
$255D == ]
$253D == =
$2521 == !
$2522 == "
$252F == _
$257C == |

$2B == +
$26 == &
$3D == =
$3F == ?
4

10 回答 10

8

我不得不做类似的事情来为 lambda 生成一个指向日志的反向链接，并做了以下骇人听闻的事情来创建链接：

const link = `https://${process.env.AWS_REGION}.console.aws.amazon.com/cloudwatch/home?region=${process.env.AWS_REGION}#logsV2:log-groups/log-group/${process.env.AWS_LAMBDA_LOG_GROUP_NAME.replace(/\//g, '$252F')}/log-events/${process.env.AWS_LAMBDA_LOG_STREAM_NAME.replace('$', '$2524').replace('[', '$255B').replace(']', '$255D').replace(/\//g, '$252F')}`
于 2020-07-05T22:19:16.620 回答
7

我的一位同事发现编码没什么特别的。它是标准的 URI百分比编码，但应用了两次 (2x)。在 javascript 中，您可以使用该encodeURIComponent函数进行测试：

let inp = 'https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/'

console.log(encodeURIComponent(inp))
console.log(encodeURIComponent(encodeURIComponent(inp)))

这段 javascript 在第二个编码阶段产生预期的输出：

https%3A%2F%2Fconsole.aws.amazon.com%2Fcloudwatch%2Fhome%3Fregion%3Dus-east-1%23logsV2%3Alog-groups%2Flog-group%2F
https%253A%252F%252Fconsole.aws.amazon.com%252Fcloudwatch%252Fhome%253Fregion%253Dus-east-1%2523logsV2%253Alog-groups%252Flog-group%252F

警告

至少有些位使用双重编码，而不是整个链接。否则所有特殊字符在双重编码后会占用4个字符，但有些仍然只占用2个字符。无论如何希望这会有所帮助；）

于 2020-10-22T14:48:47.367 回答
6

首先，我要感谢其他人提供的线索。进一步了解如何构建 Log Insights 链接的完整说明。

总的来说，它只是一个像这样工作的对象结构的奇怪编码组合：

  • 之后的部分?queryDetail=是对象表示，{}由~()

  • 对象被遍历到原始值，后者被转换如下：

    • encodeURIComponent(value)以便所有特殊字符都转换为%xx
    • replace(/%/g, "*")以便此编码不受顶级编码的影响
    • 如果值类型是string- 它以不匹配的单引号为前缀

    为了显示：

    "Hello world" -> "Hello%20world" -> "Hello*20world" -> "'Hello*20world"

  • 转换后的基元数组使用~并放入~()构造中

然后，在完成基元转换后 - 使用“~”连接对象。

在那个字符串之后是escape()d （注意 notencodeURIComponent()被调用，因为它不会~在 JS 中转换）。

之后?queryDetail=添加。

最后，这个字符串 us encodeURIComponent()ed 并作为顶部的樱桃 -%被替换为$.

让我们看看它在实践中是如何工作的。假设这些是我们的查询参数：

const expression = `fields @timestamp, @message
    | filter @message not like 'example'
    | sort @timestamp asc
    | limit 100`;

const logGroups = ["/application/sample1", "/application/sample2"];

const queryParameters = {
  end: 0,
  start: -3600,
  timeType: "RELATIVE",
  unit: "seconds",
  editorString: expression,
  isLiveTrail: false,
  source: logGroups,
};

首先对原语进行转换：

const expression = "'fields*20*40timestamp*2C*20*40message*0A*20*20*20*20*7C*20filter*20*40message*20not*20like*20'example'*0A*20*20*20*20*7C*20sort*20*40timestamp*20asc*0A*20*20*20*20*7C*20limit*20100";

const logGroups = ["'*2Fapplication*2Fsample1", "'*2Fapplication*2Fsample2"];

const queryParameters = {
  end: 0,
  start: -3600,
  timeType: "'RELATIVE",
  unit: "'seconds",
  editorString: expression,
  isLiveTrail: false,
  source: logGroups,
};

然后，对象被连接使用~，所以我们有对象表示字符串：

const objectString = "~(end~0~start~-3600~timeType~'RELATIVE~unit~'seconds~editorString~'fields*20*40timestamp*2C*20*40message*0A*20*20*20*20*7C*20filter*20*40message*20not*20like*20'example'*0A*20*20*20*20*7C*20sort*20*40timestamp*20asc*0A*20*20*20*20*7C*20limit*20100~isLiveTrail~false~source~(~'*2Fapplication*2Fsample1~'*2Fapplication*2Fsample2))"

现在我们escape()：

const escapedObject = "%7E%28end%7E0%7Estart%7E-3600%7EtimeType%7E%27RELATIVE%7Eunit%7E%27seconds%7EeditorString%7E%27fields*20*40timestamp*2C*20*40message*0A*20*20*20*20*7C*20filter*20*40message*20not*20like*20%27example%27*0A*20*20*20*20*7C*20sort*20*40timestamp*20asc*0A*20*20*20*20*7C*20limit*20100%7EisLiveTrail%7Efalse%7Esource%7E%28%7E%27*2Fapplication*2Fsample1%7E%27*2Fapplication*2Fsample2%29%29"

现在我们附加?queryDetail=前缀：

const withQueryDetail = "?queryDetail=%7E%28end%7E0%7Estart%7E-3600%7EtimeType%7E%27RELATIVE%7Eunit%7E%27seconds%7EeditorString%7E%27fields*20*40timestamp*2C*20*40message*0A*20*20*20*20*7C*20filter*20*40message*20not*20like*20%27example%27*0A*20*20*20*20*7C*20sort*20*40timestamp*20asc*0A*20*20*20*20*7C*20limit*20100%7EisLiveTrail%7Efalse%7Esource%7E%28%7E%27*2Fapplication*2Fsample1%7E%27*2Fapplication*2Fsample2%29%29"

最后我们对其进行 URLencode 并替换%为$and vois la：

const result = "$3FqueryDetail$3D$257E$2528end$257E0$257Estart$257E-3600$257EtimeType$257E$2527RELATIVE$257Eunit$257E$2527seconds$257EeditorString$257E$2527fields*20*40timestamp*2C*20*40message*0A*20*20*20*20*7C*20filter*20*40message*20not*20like*20$2527example$2527*0A*20*20*20*20*7C*20sort*20*40timestamp*20asc*0A*20*20*20*20*7C*20limit*20100$257EisLiveTrail$257Efalse$257Esource$257E$2528$257E$2527*2Fapplication*2Fsample1$257E$2527*2Fapplication*2Fsample2$2529$2529"

当然也可以执行反向操作。

这就是所有人。玩得开心，小心点，尽量避免自己做这种奇怪的事情。:)

于 2021-08-03T15:16:09.073 回答
3

我基于 @isaias-b 答案的完整 Javascript 解决方案，它还在日志上添加了时间戳过滤器：

const logBaseUrl = 'https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group';
const encode = text => encodeURIComponent(text).replace(/%/g, '$');
const awsEncode = text => encodeURIComponent(encodeURIComponent(text)).replace(/%/g, '$');
const encodeTimestamp = timestamp => encode('?start=') + awsEncode(new Date(timestamp).toJSON());
const awsLambdaLogBaseUrl = `${logBaseUrl}/${awsEncode('/aws/lambda/')}`;
const logStreamUrl = (logGroup, logStream, timestamp) =>
  `${awsLambdaLogBaseUrl}${logGroup}/log-events/${awsEncode(logStream)}${timestamp ? encodeTimestamp(timestamp) : ''}`;
于 2021-01-09T09:59:01.540 回答
2

我创建了一些似乎满足 CloudWatch URL 解析器的 Ruby 代码。我不确定为什么你必须双重逃避某些东西，然后%用$其他东西代替。我猜这背后有一些原因，但我想不出一个好的方法来做到这一点，所以我只是蛮力强迫它。如果你有更好的东西，或者知道他们为什么这样做，请添加评论。

注意：filter我测试的有点基本，如果你真的喜欢它，我不确定可能需要改变什么。

# Basic URL that is the same across all requests
url = 'https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/'

# CloudWatch log group
log_group = '/aws/my/log/group'

# Either specify the instance you want to search or leave it out to search all instances
instance = '/log-events/i-xxxxxxxxxxxx'
 OR
instance = '/log-events'

# The filter to apply.
filter = '[incoming_ip, user_name, user_ip , timestamp, request, status_code = 5*, bytes, url, user_agent]'

# Start time.  There might be an End time as well but my queries haven't used 
# that yet so I'm not sure how it's formatted.  It should be pretty similar
# though.
hours = 48
start = "&start=-#{hours*60*60*1000}"

# This will get you the final URL
final = url + CGI.escape(CGI.escape(log_group)) + instance + '$3FfilterPattern$3D' + CGI.escape(CGI.escape(filter)).gsub('%','$') + CGI.escape(start).gsub('%','$')
于 2020-03-23T17:30:23.277 回答
2

我最近想生成 cloudwatch 洞察 URL 时遇到了这个问题。打字稿版本如下：

export function getInsightsUrl(
  start: Date,
  end: Date,
  query: string,
  sourceGroup: string,
  region = "us-east-1"
) {
  const p = (m: string) => escape(m);

  // encodes inner values
  const s = (m: string) => escape(m).replace(/\%/gi, "*");

  const queryDetail =
    p(`~(end~'`) +
    s(end.toISOString()) +
    p(`~start~'`) +
    s(start.toISOString()) +
    p(`~timeType~'ABSOLUTE~tz~'UTC~editorString~'`) +
    s(query) +
    p(`~isLiveTail~false~queryId~'`) +
    s(v4()) +
    p(`~source~(~'`) +
    s(sourceGroup) +
    p(`))`);

  return (
    `https://console.aws.amazon.com/cloudwatch/home?region=${region}#logsV2:logs-insights` +
    escape("?queryDetail=" + queryDetail).replace(/\%/gi, "$")
  );
}

Github GIST

于 2021-03-27T12:06:34.347 回答
1

有点晚了，但这是一个 python 实现

def get_cloud_watch_search_url(search, log_group, log_stream, region=None,):
    """Return a properly formatted url string for search cloud watch logs

    search = "{$.message: "You are amazing"}
    log_group = Is the group of message you want to search
    log_stream = The stream of logs to search
    """

    url = f'https://{region}.console.aws.amazon.com/cloudwatch/home?region={region}'

    def aws_encode(value):
        """The heart of this is that AWS likes to quote things twice with some substitution"""
        value = urllib.parse.quote_plus(value)
        value = re.sub(r"\+", " ", value)
        return re.sub(r"%", "$", urllib.parse.quote_plus(value))

    bookmark = '#logsV2:log-groups'
    bookmark += '/log-group/' + aws_encode(log_group)
    bookmark += "/log-events/" + log_stream
    bookmark += re.sub(r"%", "$", urllib.parse.quote("?filterPattern="))
    bookmark += aws_encode(search)

    return url + bookmark

然后，这使您可以快速验证它。

>>> real = 'https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#logsV2:log-groups/log-group/$252Fapp$252Fdjango/log-events/production$3FfilterPattern$3D$257B$2524.msg$253D$2522$2525s$2525s+messages+to+$2525s+pk$253D$2525d...$2522$257D'
>>> constructed = get_cloud_watch_search_url(None, search='{$.msg="%s%s messages to %s pk=%d..."}', log_group='/app/django', log_stream='production', region='us-west-2')
>>> real == constructed
True
于 2021-03-10T20:05:05.047 回答
1

基于@Pål Brattberg 回答的 Python 解决方案：

cloudwatch_log_template = "https://{AWS_REGION}.console.aws.amazon.com/cloudwatch/home?region={AWS_REGION}#logsV2:log-groups/log-group/{LOG_GROUP_NAME}/log-events/{LOG_STREAM_NAME}"
log_url = cloudwatch_log_template.format(
    AWS_REGION=AWS_REGION, LOG_GROUP_NAME=CLOUDWATCH_LOG_GROUP, LOG_STREAM_NAME=LOG_STREAM_NAME
)

如果您使用了任何字符，请确保首先替换非法字符（请参阅 OP）。

于 2021-02-10T13:54:28.957 回答
1

我最近想生成 cloudwatch 洞察 URL 时遇到了这个问题。PHP版本如下：

  <?php
  function getInsightsUrl($region = 'ap-northeast-1') {

    // https://stackoverflow.com/questions/67734825/why-is-laravels-carbon-toisostring-different-from-javascripts-toisostring
    $start = now()->subMinutes(2)->format('Y-m-d\TH:i:s.v\Z');
    $end = now()->addMinutes(2)->format('Y-m-d\TH:i:s.v\Z');

    $filter = 'INFO';

    $logStream = 'xxx_backend_web';
    $sourceGroup = '/ecs/xxx_backend_prod';
    // $sourceGroup = '/aws/ecs/xxx_backend~\'/ecs/xxx_backend_dev'; // multiple source group

    $query =
        "fields @timestamp, @message \n" .
        "| sort @timestamp desc\n" .
        "| filter @logStream like '$logStream'\n" .
        "| filter @message like '$filter'\n" .
        "| limit 20";

    $queryDetail = urlencode(
        ("~(end~'") .
        ($end) .
        ("~start~'") .
        ($start) .
        ("~timeType~'ABSOLUTE~tz~'Local~editorString~'") .
        ($query) .
        ("~isLiveTail~false~queryId~'") .
        ("~source~(~'") .
        ($sourceGroup) .
        ("))")
    );

    $queryDetail = preg_replace('/\%/', '$', urlencode("?queryDetail=" . $queryDetail));

    return
        "https://console.aws.amazon.com/cloudwatch/home?region=${region}#logsV2:logs-insights"
        . $queryDetail;
}
于 2021-07-05T08:24:36.577 回答
0

由于 Python 贡献与日志组相关，而不是日志洞察，这是我的贡献。我想我可以用内部函数做得更好，但这是一个很好的起点：

from datetime import datetime, timedelta
import re
from urllib.parse import quote
 
def get_aws_cloudwatch_log_insights(query_parameters, aws_region):
    def quote_string(input_str):
        return f"""{quote(input_str, safe="~()'*").replace('%', '*')}"""


    def quote_list(input_list):
        quoted_list = ""
        for item in input_list:
            if isinstance(item, str):
                item = f"'{item}"

            quoted_list += f"~{item}"
        return f"({quoted_list})"


    params = []
    for key, value in query_parameters.items():
        if key == "editorString":
            value = "'" + quote(value)
            value = value.replace('%', '*')
        elif isinstance(value, str):
            value = "'" + value
        if isinstance(value, bool):
            value = str(value).lower()
        elif isinstance(value, list):
            value = quote_list(value)
        params += [key, str(value)]

    object_string = quote_string("~(" + "~".join(params) + ")")
    scaped_object = quote(object_string, safe="*").replace("~", "%7E")
    with_query_detail = "?queryDetail=" + scaped_object
    result = quote(with_query_detail, safe="*").replace("%", "$")

    final_url = f"https://{aws_region}.console.aws.amazon.com/cloudwatch/home?region={aws_region}#logsV2:logs-insights{result}"

    return final_url

例子：

aws_region = "eu-west-1"

query = """fields @timestamp, @message
| filter @message not like 'example'
| sort @timestamp asc
| limit 100"""

log_groups = ["/application/sample1", "/application/sample2"]

query_parameters = {
  "end": datetime.utcnow().isoformat(timespec='milliseconds') + "Z",
  "start": (datetime.utcnow() - timedelta(days=2)).isoformat(timespec='milliseconds') + "Z",
  "timeType": "ABSOLUTE",
  "unit": "seconds",
  "editorString": query,
  "isLiveTrail": False,
  "source": log_groups,
}

print(get_aws_cloudwatch_log_insights(query_parameters, aws_region))
于 2021-12-28T11:40:30.467 回答