Monospaced |
Used for commands, HTTP request and responses and code blocks. |
|---|---|
<Monospaced> |
User entered values. |
[Monospaced] |
Optional values. When the value is not specified, the default value is used. |
| 斜体 | 重要短语和词汇。 |
HTTP REST API 支持完整的HDFS FileSystem/FileContext接口。下一节将展示这些操作及其对应的FileSystem/FileContext方法。HTTP Query Parameter Dictionary章节详细说明了参数的默认值和有效值等细节。
OPEN (参见 FileSystem.open)GETFILESTATUS (参见 FileSystem.getFileStatus)LISTSTATUS (参见 FileSystem.listStatus)LISTSTATUS_BATCH (参见 FileSystem.listStatusIterator)GETCONTENTSUMMARY (参见 FileSystem.getContentSummary)GETQUOTAUSAGE (参见 FileSystem.getQuotaUsage)GETFILECHECKSUM (参见 FileSystem.getFileChecksum)GETHOMEDIRECTORY (参见 FileSystem.getHomeDirectory)GETDELEGATIONTOKEN (参见 FileSystem.getDelegationToken)GETTRASHROOT (参见 FileSystem.getTrashRoot)GETXATTRS (参见 FileSystem.getXAttr)GETXATTRS (参见 FileSystem.getXAttrs)GETXATTRS (参见 FileSystem.getXAttrs)LISTXATTRS (参见 FileSystem.listXAttrs)CHECKACCESS (参见 FileSystem.access)GETALLSTORAGEPOLICY (参见 FileSystem.getAllStoragePolicies)GETSTORAGEPOLICY (参见 FileSystem.getStoragePolicy)GETSNAPSHOTDIFFGETSNAPSHOTDIFFLISTINGGETSNAPSHOTTABLEDIRECTORYLISTGETSNAPSHOTLISTGETFILEBLOCKLOCATIONS (参见 FileSystem.getFileBlockLocations)GETECPOLICY (参见 HDFSErasureCoding.getErasureCodingPolicy)GETSERVERDEFAULTS (参见 FileSystem.getServerDefaults)GETLINKTARGET (参见 FileSystem.getLinkTarget)GETFILELINKSTATUS (参见 FileSystem.getFileLinkStatus)GETSTATUS (参见 FileSystem.getStatus)GETECPOLICIESGETECCODECSGETTRASHROOTS (参见 FileSystem.getTrashRoots)CREATE (参见 FileSystem.create)MKDIRS (参见 FileSystem.mkdirs)CREATESYMLINK (参见 FileContext.createSymlink)RENAME (参见 FileSystem.rename)SETREPLICATION (参见 FileSystem.setReplication)SETOWNER (参见 FileSystem.setOwner)SETPERMISSION (参见 FileSystem.setPermission)SETTIMES (参见 FileSystem.setTimes)RENEWDELEGATIONTOKEN (参见 DelegationTokenAuthenticator.renewDelegationToken)CANCELDELEGATIONTOKEN (参见 DelegationTokenAuthenticator.cancelDelegationToken)ALLOWSNAPSHOTDISALLOWSNAPSHOTCREATESNAPSHOT (参见 FileSystem.createSnapshot)RENAMESNAPSHOT (参见 FileSystem.renameSnapshot)SETXATTR (参见 FileSystem.setXAttr)REMOVEXATTR (参见 FileSystem.removeXAttr)SETSTORAGEPOLICY (参见 FileSystem.setStoragePolicy)SATISFYSTORAGEPOLICY (参见 ArchivalStorage.satisfyStoragePolicy)ENABLEECPOLICY (参见 HDFSErasureCoding.enablePolicy)DISABLEECPOLICY (参见 HDFSErasureCoding.disablePolicy)SETECPOLICY (参见 HDFSErasureCoding.setErasureCodingPolicy)APPEND (参见 FileSystem.append)CONCAT (参见 FileSystem.concat)TRUNCATE (参见 FileSystem.truncate)UNSETSTORAGEPOLICY (参见 FileSystem.unsetStoragePolicy)UNSETECPOLICY (参见 HDFSErasureCoding.unsetErasureCodingPolicy)DELETE (参见 FileSystem.delete)DELETESNAPSHOT (参见 FileSystem.deleteSnapshot)WebHDFS 的文件系统方案是“webhdfs://”。WebHDFS 文件系统 URI 的格式如下。
webhdfs://<HOST>:<HTTP_PORT>/<PATH>
上述WebHDFS URI对应以下HDFS URI。
hdfs://<HOST>:<RPC_PORT>/<PATH>
在REST API中，路径前缀会插入"/webhdfs/v1"并在末尾附加查询参数。因此，对应的HTTP URL具有以下格式。
http://<HOST>:<HTTP_PORT>/webhdfs/v1/<PATH>?op=...
注意，如果WebHDFS启用了SSL安全连接，那么协议方案应为“swebhdfs://”。
swebhdfs://<HOST>:<HTTP_PORT>/<PATH>
以下是WebHDFS的HDFS配置选项。
| 属性名称 | 描述 |
|---|---|
dfs.web.authentication.kerberos.principal |
The HTTP Kerberos principal used by Hadoop-Auth in the HTTP endpoint. The HTTP Kerberos principal MUST start with ‘HTTP/’ per Kerberos HTTP SPNEGO specification. A value of “*” will use all HTTP principals found in the keytab. |
dfs.web.authentication.kerberos.keytab |
The Kerberos keytab file with the credentials for the HTTP Kerberos principal used by Hadoop-Auth in the HTTP endpoint. |
dfs.webhdfs.socket.connect-timeout |
How long to wait for a connection to be established before failing. Specified as a time duration, ie numerical value followed by a units symbol, eg 2m for two minutes. Defaults to 60s. |
dfs.webhdfs.socket.read-timeout |
How long to wait for data to arrive before failing. Defaults to 60s. |
当安全功能处于关闭状态时，认证用户即为user.name查询参数中指定的用户名。如果未设置user.name参数，服务器可能会将认证用户设为默认的Web用户（如果存在），或者返回错误响应。
当安全机制启用时，认证将通过Hadoop委托令牌或Kerberos SPNEGO进行。如果在delegation查询参数中设置了令牌，则认证用户为令牌中编码的用户。如果未设置delegation参数，则用户将通过Kerberos SPNEGO进行认证。
以下是使用curl命令行工具的示例。
安全关闭时的认证：
curl -i "http://: /webhdfs/v1/ ?[user.name= &]op=..."
在启用安全认证时使用Kerberos SPNEGO进行身份验证：
curl -i --negotiate -u : "http://: /webhdfs/v1/ ?op=..."
在启用安全认证时使用Hadoop委托令牌进行身份验证：
curl -i "http://: /webhdfs/v1/ ?delegation= &op=..."
另请参阅：Authentication for Hadoop HTTP web-consoles
此外，WebHDFS在客户端支持OAuth2协议。目前Namenode和Datanode尚不支持使用OAuth2的客户端，但其他实现了WebHDFS REST接口的后端系统可能支持。
WebHDFS默认支持两种OAuth2授权码模式（用户提供的刷新令牌与访问令牌，或用户提供的凭证），并提供了可插拔机制用于根据OAuth2 RFC规范实现其他OAuth2认证方式或自定义认证。当使用任一内置的授权码机制时，WebHDFS客户端会在必要时自动刷新访问令牌。
OAuth2 应仅针对未使用 Kerberos SPENGO 运行的客户端启用。
| OAuth2 code grant mechanism | Description | Value of dfs.webhdfs.oauth2.access.token.provider that implements code grant |
|---|---|---|
| 授权码授权 | 用户提供初始访问令牌和刷新令牌，这些令牌随后分别用于验证WebHDFS请求和获取替换访问令牌。 | org.apache.hadoop.hdfs.web.oauth2.ConfRefreshTokenBasedAccessTokenProvider |
| 客户端凭证授权 | 用户提供凭证用于获取访问令牌，这些令牌随后用于验证WebHDFS请求。 | org.apache.hadoop.hdfs.web.oauth2.ConfCredentialBasedAccessTokenProvider |
以下属性控制OAuth2认证。
| OAuth2 相关属性 | 描述 |
|---|---|
dfs.webhdfs.oauth2.enabled |
Boolean to enable/disable OAuth2 authentication |
dfs.webhdfs.oauth2.access.token.provider |
Class name of an implementation of org.apache.hadoop.hdfs.web.oauth.AccessTokenProvider. Two are provided with the code, as described above, or the user may specify a user-provided implementation. The default value for this configuration key is the ConfCredentialBasedAccessTokenProvider implementation. |
dfs.webhdfs.oauth2.client.id |
Client id used to obtain access token with either credential or refresh token |
dfs.webhdfs.oauth2.refresh.url |
URL against which to post for obtaining bearer token with either credential or refresh token |
dfs.webhdfs.oauth2.access.token |
(required if using ConfRefreshTokenBasedAccessTokenProvider) Initial access token with which to authenticate |
dfs.webhdfs.oauth2.refresh.token |
(required if using ConfRefreshTokenBasedAccessTokenProvider) Initial refresh token to use to obtain new access tokens |
dfs.webhdfs.oauth2.refresh.token.expires.ms.since.epoch |
(required if using ConfRefreshTokenBasedAccessTokenProvider) Access token expiration measured in milliseconds since Jan 1, 1970. Note this is a different value than provided by OAuth providers and has been munged as described in interface to be suitable for a client application |
dfs.webhdfs.oauth2.credential |
(required if using ConfCredentialBasedAccessTokenProvider). Credential used to obtain initial and subsequent access tokens. |
要使用SWebHDFS文件系统（即使用swebhdfs协议），需要在客户端指定一个SSL配置文件。该文件必须包含以下3个参数：
| SSL属性 | 描述 |
|---|---|
ssl.client.truststore.location |
The local-filesystem location of the trust-store file, containing the certificate for the NameNode. |
ssl.client.truststore.type |
(Optional) The format of the trust-store file. |
ssl.client.truststore.password |
(Optional) Password for the trust-store file. |
以下是一个SSL配置文件的示例（ssl-client.xml）：
<configuration>
<property>
<name>ssl.client.truststore.location</name>
<value>/work/keystore.jks</value>
<description>Truststore to be used by clients. Must be specified.</description>
</property>
<property>
<name>ssl.client.truststore.password</name>
<value>changeme</value>
<description>Optional. Default value is "".</description>
</property>
<property>
<name>ssl.client.truststore.type</name>
<value>jks</value>
<description>Optional. Default value is "jks".</description>
</property>
</configuration>
SSL配置文件必须位于客户端程序的类路径中，且文件名需要在core-site.xml中指定：
<property>
<name>hadoop.ssl.client.conf</name>
<value>ssl-client.xml</value>
<description>
Resource file from which ssl client keystore information will be extracted.
This file is looked up in the classpath, typically it should be in Hadoop
conf/ directory. Default value is "ssl-client.xml".
</description>
</property>
当代理用户功能启用时，代理用户P可以代表另一个用户U提交请求。除非在认证中提供了委托令牌，否则必须在doas查询参数中指定用户U的用户名。在这种情况下，用户P和U的信息都必须编码在委托令牌中。
当安全关闭时的代理请求：
curl -i "http://: /webhdfs/v1/ ?[user.name= &]doas= &op=..."
当安全认证开启时，使用Kerberos SPNEGO的代理请求：
curl -i --negotiate -u : "http://: /webhdfs/v1/ ?doas= &op=..."
在安全模式下使用Hadoop委托令牌的代理请求：
curl -i "http://: /webhdfs/v1/ ?delegation= &op=..."
WebHDFS支持一个可选的、可配置的跨站请求伪造(CSRF)防护机制。启用该功能后，发送至NameNode或DataNode的WebHDFS HTTP请求必须包含一个自定义HTTP头部。通过配置属性可以调整受保护的具体HTTP方法以及HTTP头部的名称。头部中发送的具体值无关紧要，只需确保存在该名称的头部即可。
启用CSRF防护还会配置WebHdfsFileSystem类发送所需的请求头。这确保当使用webhdfs: URI时，像hdfs dfs和hadoop distcp这样的CLI命令能继续正常工作。
启用CSRF防护还会设置NameNode网页界面发送所需的标头。在启用CSRF防护并重启NameNode后，现有NameNode网页界面的用户需要刷新浏览器以重新加载页面并获取新配置。
以下属性控制CSRF防护。
| 属性 | 描述 | 默认值 |
|---|---|---|
dfs.webhdfs.rest-csrf.enabled |
If true, then enables WebHDFS protection against cross-site request forgery (CSRF). The WebHDFS client also uses this property to determine whether or not it needs to send the custom CSRF prevention header in its HTTP requests. | false |
dfs.webhdfs.rest-csrf.custom-header |
The name of a custom header that HTTP requests must send when protection against cross-site request forgery (CSRF) is enabled for WebHDFS by setting dfs.webhdfs.rest-csrf.enabled to true. The WebHDFS client also uses this property to determine whether or not it needs to send the custom CSRF prevention header in its HTTP requests. | X-XSRF-HEADER |
dfs.webhdfs.rest-csrf.methods-to-ignore |
A comma-separated list of HTTP methods that do not require HTTP requests to include a custom header when protection against cross-site request forgery (CSRF) is enabled for WebHDFS by setting dfs.webhdfs.rest-csrf.enabled to true. The WebHDFS client also uses this property to determine whether or not it needs to send the custom CSRF prevention header in its HTTP requests. | GET,OPTIONS,HEAD,TRACE |
dfs.webhdfs.rest-csrf.browser-useragents-regex |
A comma-separated list of regular expressions used to match against an HTTP request’s User-Agent header when protection against cross-site request forgery (CSRF) is enabled for WebHDFS by setting dfs.webhdfs.reset-csrf.enabled to true. If the incoming User-Agent matches any of these regular expressions, then the request is considered to be sent by a browser, and therefore CSRF prevention is enforced. If the request’s User-Agent does not match any of these regular expressions, then the request is considered to be sent by something other than a browser, such as scripted automation. In this case, CSRF is not a potential attack vector, so the prevention is not enforced. This helps achieve backwards-compatibility with existing automation that has not been updated to send the CSRF prevention header. | ^Mozilla.*,^Opera.* |
dfs.datanode.httpserver.filter.handlers |
Comma separated list of Netty servlet-style filter handlers to inject into the Datanode WebHDFS I/O path | org.apache.hadoop.hdfs.server.datanode.web.RestCsrfPreventionFilterHandler |
以下是一个使用curl命令的示例，其中通过-H选项在请求中包含自定义头部。
curl -i -L -X PUT -H 'X-XSRF-HEADER: ""' 'http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=CREATE'
WebHDFS支持可选的、可配置的重试策略，用于处理可能超时的大文件弹性复制，或在复制过程中可能发生故障转移的高可用性集群之间复制文件。
以下属性控制WebHDFS的重试和故障转移策略。
| 属性 | 描述 | 默认值 |
|---|---|---|
dfs.http.client.retry.policy.enabled |
If “true”, enable the retry policy of WebHDFS client. If “false”, retry policy is turned off. | false |
dfs.http.client.retry.policy.spec |
Specify a policy of multiple linear random retry for WebHDFS client, e.g. given pairs of number of retries and sleep time (n0, t0), (n1, t1), …, the first n0 retries sleep t0 milliseconds on average, the following n1 retries sleep t1 milliseconds on average, and so on. | 10000,6,60000,10 |
dfs.http.client.failover.max.attempts |
Specify the max number of failover attempts for WebHDFS client in case of network exception. | 15 |
dfs.http.client.retry.max.attempts |
Specify the max number of retry attempts for WebHDFS client, if the difference between retried attempts and failovered attempts is larger than the max number of retry attempts, there will be no more retries. | 10 |
dfs.http.client.failover.sleep.base.millis |
Specify the base amount of time in milliseconds upon which the exponentially increased sleep time between retries or failovers is calculated for WebHDFS client. | 500 |
dfs.http.client.failover.sleep.max.millis |
Specify the upper bound of sleep time in milliseconds between retries or failovers for WebHDFS client. | 15000 |
可以通过WebHDFS协议控制数据的单向传输，仅允许从不安全网络写入数据。要实现此功能，必须确保dfs.datanode.httpserver.filter.handlers包含org.apache.hadoop.hdfs.server.datanode.web.HostRestrictingAuthorizationFilterHandler。HostRestrictingAuthorizationFilter的配置通过以下属性进行控制。
| 属性 | 描述 | 默认值 |
|---|---|---|
dfs.datanode.httpserver.filter.handlers |
Comma separated list of Netty servlet-style filter handlers to inject into the Datanode WebHDFS I/O path | org.apache.hadoop.hdfs.server.datanode.web.RestCsrfPreventionFilterHandler |
dfs.web.authentication.host.allow.rules |
Rules allowing users to read files in the format of user,network/bits,path glob newline or |-separated. Use * for a wildcard of all users or network/bits. |
nothing - defaults to no one may read via WebHDFS |
步骤1：提交一个HTTP PUT请求，不自动跟随重定向且不发送文件数据。