Table Of Content

Cluster Management Guide ImportantNotice ©2010-2021Cloudera,Inc.Allrightsreserved. Cloudera,theClouderalogo,andanyotherproductor servicenamesorsloganscontainedinthisdocumentaretrademarksofClouderaand itssuppliersorlicensors,andmaynotbecopied,imitatedorused,inwholeorinpart, withoutthepriorwrittenpermissionofClouderaortheapplicabletrademarkholder.If thisdocumentationincludescode,includingbutnotlimitedto,codeexamples,Cloudera makesthisavailabletoyouunderthetermsoftheApacheLicense,Version2.0,including anyrequirednotices.AcopyoftheApacheLicenseVersion2.0,includinganynotices, isincludedherein.AcopyoftheApacheLicenseVersion2.0canalsobefoundhere: https://opensource.org/licenses/Apache-2.0 HadoopandtheHadoopelephantlogoaretrademarksoftheApacheSoftware Foundation.Allothertrademarks,registeredtrademarks,productnamesandcompany namesorlogosmentionedinthisdocumentarethepropertyoftheirrespectiveowners. Referencetoanyproducts,services,processesorotherinformation,bytradename, trademark,manufacturer,supplierorotherwisedoesnotconstituteorimply endorsement,sponsorshiporrecommendationthereofbyus. Complyingwithallapplicablecopyrightlawsistheresponsibilityoftheuser.Without limitingtherightsundercopyright,nopartofthisdocumentmaybereproduced,stored inorintroducedintoaretrievalsystem,ortransmittedinanyformorbyanymeans (electronic,mechanical,photocopying,recording,orotherwise),orforanypurpose, withouttheexpresswrittenpermissionofCloudera. Clouderamayhavepatents,patentapplications,trademarks,copyrights,orother intellectualpropertyrightscoveringsubjectmatterinthisdocument.Exceptasexpressly providedinanywrittenlicenseagreementfromCloudera,thefurnishingofthisdocument doesnotgiveyouanylicensetothesepatents,trademarkscopyrights,orother intellectualproperty.ForinformationaboutpatentscoveringClouderaproducts,see http://tiny.cloudera.com/patents. Theinformationinthisdocumentissubjecttochangewithoutnotice.Clouderashall notbeliableforanydamagesresultingfromtechnicalerrorsoromissionswhichmay bepresentinthisdocument,orfromuseofthisdocument. Cloudera,Inc. 395PageMillRoad PaloAlto,CA94306 [email protected] US:1-888-789-1488 Intl:1-650-362-0488 www.cloudera.com ReleaseInformation Version:ClouderaEnterprise6.3.x Date:September30,2021 Table of Contents About Cloudera Administration..............................................................................14 Cloudera Manager.................................................................................................15 Learn about Cloudera Manager ........................................................................................................................17 Installing and Upgrading....................................................................................................................................17 Managing CDH using Cloudera Manager...........................................................................................................17 MonitoringCDH using Cloudera Manager.........................................................................................................17 ManagingCDHusingtheClouderaManagerAPI...............................................................................................17 Cloudera Manager Admin Console....................................................................................................................18 StartingandLoggingintotheAdminConsole......................................................................................................................20 ClouderaManagerAdminConsoleHomePage...................................................................................................................20 DisplayingClouderaManagerDocumentation....................................................................................................................25 Automatic Logout................................................................................................................................................................25 ClouderaManagerFrequentlyAskedQuestions................................................................................................26 GeneralQuestions................................................................................................................................................................26 Cloudera Manager API.......................................................................................................................................28 BackingUpandRestoringtheClouderaManagerConfiguration.......................................................................................30 UsingtheClouderaManagerAPIforClusterAutomation...................................................................................................31 ClouderaManagerAdministration.....................................................................................................................33 Starting,Stopping,andRestartingtheClouderaManagerServer.......................................................................................33 ConfiguringClouderaManagerServerPorts.......................................................................................................................33 MovingtheClouderaManagerServertoaNewHost.........................................................................................................34 MigratingfromtheClouderaManagerEmbeddedPostgreSQLDatabaseServertoanExternalPostgreSQLDatabase .......................................................................................................................................................................................35 MigratingfromtheClouderaManagerExternalPostgreSQLDatabaseServertoaMySQL/OracleDatabaseServer........41 ManagingtheClouderaManagerServerLog......................................................................................................................44 ClouderaManagerAgents...................................................................................................................................................44 ConfiguringNetworkSettings..............................................................................................................................................50 Managing Licenses..............................................................................................................................................................50 SendingUsageandDiagnosticDatatoCloudera................................................................................................................57 ExportingandImportingClouderaManagerConfiguration................................................................................................61 BackingUpClouderaManager............................................................................................................................................61 OtherClouderaManagerTasksandSettings.......................................................................................................................66 ClouderaManagementService............................................................................................................................................67 ExtendingClouderaManager.............................................................................................................................72 Cluster Configuration Overview..............................................................................73 ModifyingConfigurationPropertiesUsingClouderaManager..........................................................................74 ChangingtheConfigurationofaServiceorRoleInstance...................................................................................................74 RestartingServicesandInstancesafterConfigurationChanges..........................................................................................78 SuppressingConfigurationandParameterValidationWarnings........................................................................................78 Autoconfiguration..............................................................................................................................................79 Autoconfiguration................................................................................................................................................................80 Role-HostPlacement............................................................................................................................................................87 Custom Configuration........................................................................................................................................88 Stale Configurations...........................................................................................................................................91 Client Configuration Files...................................................................................................................................93 HowClientConfigurationsareDeployed.............................................................................................................................93 DownloadingClientConfigurationFiles...............................................................................................................................94 ManuallyRedeployingClientConfigurationFiles................................................................................................................94 ViewingandRevertingConfigurationChanges..................................................................................................94 ViewingConfigurationChanges...........................................................................................................................................94 Reverting Configuration Changes........................................................................................................................................95 ExportingandImportingClouderaManagerConfiguration...............................................................................95 ClouderaManagerConfigurationPropertiesReference....................................................................................96 Managing Clusters..................................................................................................97 Adding and Deleting Clusters.............................................................................................................................97 AddingaClusterUsingNewHosts.......................................................................................................................................97 AddingaClusterUsingCurrentlyManagedHosts.............................................................................................................101 DeletingaCluster...............................................................................................................................................................104 Starting,Stopping,Refreshing,andRestartingaCluster..................................................................................104 PausingaClusterinAWS..................................................................................................................................106 ShuttingDownandStartingUptheCluster.......................................................................................................................106 Considerations after Restart..............................................................................................................................................107 Renaming a Cluster..........................................................................................................................................108 Cluster-WideConfiguration..............................................................................................................................108 VirtualPrivateClusters andCloudera SDX.......................................................................................................108 Overview............................................................................................................................................................................109 AdvantagesofSeparatingComputeandDataResources..................................................................................................109 Architecture.......................................................................................................................................................................109 PerformanceTradeOffs.....................................................................................................................................................111 UsingVirtualPrivateClustersinYourApplications............................................................................................................111 AddingaComputeClusterandDataContext....................................................................................................................111 ImprovementsforVirtualPrivateClustersinCDH6.3.......................................................................................................112 CompatibilityConsiderationsforVirtualPrivateClusters.................................................................................................113 Tutorial:UsingImpala,HiveandHuewithVirtualPrivateClusters...................................................................................116 NetworkingConsiderationsforVirtualPrivateClusters.....................................................................................................127 Managing Services...........................................................................................................................................132 ManagingtheHBaseService.............................................................................................................................................132 Managing HDFS.................................................................................................................................................................132 ManagingApacheHiveinCDH..........................................................................................................................................170 Managing Hue...................................................................................................................................................................170 Managing Impala..............................................................................................................................................................174 ManagingKey-ValueStoreIndexer....................................................................................................................................185 Managing Kudu.................................................................................................................................................................186 Managing Solr...................................................................................................................................................................187 ManagingSpark.................................................................................................................................................................192 ManagingtheSqoop1Client.............................................................................................................................................193 ManagingYARN(MRv2)andMapReduce(MRv1).............................................................................................................196 ManagingZooKeeper.........................................................................................................................................................218 ConfiguringServicestoUsetheGPLExtrasParcel.............................................................................................................221 Managing Hosts...................................................................................................223 The Status Tab..................................................................................................................................................223 The Configuration Tab......................................................................................................................................224 TheRolesandDisksOverviewTabs..................................................................................................................224 TheTemplatesTab............................................................................................................................................224 TheParcelsTab.................................................................................................................................................224 Viewing Host Details........................................................................................................................................224 Status.................................................................................................................................................................................225 Processes...........................................................................................................................................................................226 Resources...........................................................................................................................................................................226 Commands.........................................................................................................................................................................227 Configuration.....................................................................................................................................................................227 Components.......................................................................................................................................................................227 Audits.................................................................................................................................................................................227 ChartsLibrary.....................................................................................................................................................................227 UsingtheHostInspector..................................................................................................................................228 RunningtheHostInspector................................................................................................................................................228 ViewingPastHostInspectorResults..................................................................................................................................228 AddingaHosttotheCluster............................................................................................................................228 UsingtheAddHostsWizardtoAddHosts.........................................................................................................................229 AddingaHostbyInstallingthePackagesUsingYourOwnMethod..................................................................................235 SpecifyingRacksforHosts................................................................................................................................235 Host Templates.................................................................................................................................................236 Creating a Host Template..................................................................................................................................................236 EditingaHostTemplate.....................................................................................................................................................236 ApplyingaHostTemplatetoaHost...................................................................................................................................237 PerformingMaintenanceonaClusterHost.....................................................................................................237 Decommissioning Hosts.....................................................................................................................................................237 Recommissioning Hosts.....................................................................................................................................................239 StoppingAlltheRolesonaHost........................................................................................................................................239 StartingAlltheRolesonaHost..........................................................................................................................................239 TuningandTroubleshootingHostDecommissioning.........................................................................................................239 MaintenanceMode............................................................................................................................................................242 ChangingHostnames........................................................................................................................................245 Deleting Hosts..................................................................................................................................................247 Moving a Host Between Clusters.....................................................................................................................248 Managing Services...............................................................................................249 AddingaService...............................................................................................................................................249 ComparingConfigurationsforaServiceBetweenClusters..............................................................................250 Add-onServices................................................................................................................................................251 Custom ServiceDescriptorFiles.........................................................................................................................................251 InstallinganAdd-onService...............................................................................................................................................251 AddinganAdd-onService..................................................................................................................................................253 UninstallinganAdd-onService..........................................................................................................................................253 Starting, Stopping, and Restarting Services.....................................................................................................253 Startingand Stopping Services..........................................................................................................................................253 Restarting a Service...........................................................................................................................................................254 Rolling Restart..................................................................................................................................................254 Aborting a Pending Command.........................................................................................................................257 Deleting Services..............................................................................................................................................257 Renaming a Service..........................................................................................................................................258 ConfiguringMaximumFileDescriptors............................................................................................................258 ExposingHadoopMetricstoGraphite.............................................................................................................258 ConfigureHadoopMetricsforGraphiteUsingClouderaManager....................................................................................259 GraphiteConfigurationSettingsPerDaemon....................................................................................................................260 ExposingHadoopMetricstoGanglia...............................................................................................................261 ConfigureHadoopMetricsforGangliaUsingClouderaManager.....................................................................................261 GangliaConfigurationSettingsPerDaemon.....................................................................................................................263 Managing Roles....................................................................................................265 Role Instances..................................................................................................................................................265 Role Groups......................................................................................................................................................268 CreatingaRoleGroup........................................................................................................................................................268 Managing Role Groups......................................................................................................................................................269 Monitoring and Diagnostics..................................................................................270 Introduction to Cloudera Manager Monitoring...............................................................................................270 TimeLine............................................................................................................................................................................271 HealthTests........................................................................................................................................................................272 ClouderaManagerAdminConsoleHomePage.................................................................................................................274 ViewingChartsforCluster,Service,Role,andHostInstances............................................................................................279 ConfiguringMonitoringSettings........................................................................................................................................280 MonitoringClusters..........................................................................................................................................287 Inspecting Network Performance....................................................................................................................289 RunningtheNetworkPerformanceInspectorFromtheClouderaManagerAdminConsole.............................................289 RunningtheNetworkPerformanceInspectorFromtheClouderaManagerAPI...............................................................291 Monitoring Services.........................................................................................................................................292 MonitoringServiceStatus..................................................................................................................................................292 ViewingServiceStatus.......................................................................................................................................................293 ViewingServiceInstanceDetails........................................................................................................................................296 ViewingRoleInstanceStatus.............................................................................................................................................298 RunningDiagnosticCommandsforRoles..........................................................................................................................299 PeriodicStacksCollection...................................................................................................................................................300 ViewingRunningandRecentCommands..........................................................................................................................301 MonitoringResourceManagement...................................................................................................................................303 Monitoring Hosts.............................................................................................................................................304 HostDetails........................................................................................................................................................................305 Host Inspector....................................................................................................................................................................308 Monitoring Activities........................................................................................................................................309 Monitoring MapReduce Jobs.............................................................................................................................................309 MonitoringImpalaQueries................................................................................................................................................316 MonitoringYARNApplications...........................................................................................................................................327 Monitoring Spark Applications..........................................................................................................................................340 Events...............................................................................................................................................................347 Viewing Events...................................................................................................................................................................348 FilteringEvents...................................................................................................................................................................348 Alerts................................................................................................................................................................349 Managing Alerts................................................................................................................................................................349 Triggers.............................................................................................................................................................356 ClouderaManagerTriggerUseCases................................................................................................................................360 Lifecycle and Security Auditing........................................................................................................................363 Viewing Audit Events.........................................................................................................................................................364 FilteringAuditEvents.........................................................................................................................................................364 DownloadingAuditEvents.................................................................................................................................................365 ChartingTime-SeriesData................................................................................................................................365 Terminology.......................................................................................................................................................................366 BuildingaChartwithTime-SeriesData.............................................................................................................................366 ConfiguringTime-SeriesQueryResults..............................................................................................................................367 UsingContext-SensitiveVariablesinCharts.......................................................................................................................367 Chart Properties.................................................................................................................................................................368 DisplayingChartDetails.....................................................................................................................................................371 Editing a Chart...................................................................................................................................................................373 SavingaChart....................................................................................................................................................................373 ObtainingTime-SeriesDataUsingtheAPI.........................................................................................................................374 Dashboards........................................................................................................................................................................374 tsquery Language..............................................................................................................................................................377 MetricAggregation............................................................................................................................................................386 Logs..................................................................................................................................................................389 Viewing Logs......................................................................................................................................................................389 Logs List.............................................................................................................................................................................389 FilteringLogs......................................................................................................................................................................390 Log Details.........................................................................................................................................................................390 ViewingtheClouderaManagerServerLog.......................................................................................................................391 ViewingtheClouderaManagerAgentLogs......................................................................................................................391 ManagingDiskSpaceforLogFiles....................................................................................................................................392 Reports.............................................................................................................................................................392 DirectoryUsageReport......................................................................................................................................................393 Disk Usage Reports............................................................................................................................................................395 Activity,Application,andQueryReports...........................................................................................................................396 TheFileBrowser.................................................................................................................................................................396 DownloadingHDFSDirectoryAccessPermissionReports..................................................................................................397 TroubleshootingClusterConfigurationandOperation....................................................................................398 SolutionstoCommonProblems.........................................................................................................................................398 Logs and Events.................................................................................................................................................................400 Monitoring Reference......................................................................................................................................400 Performance Management...................................................................................401 OptimizingPerformanceinCDH.......................................................................................................................401 ChoosingandConfiguringDataCompression..................................................................................................405 ConfiguringData Compression..........................................................................................................................................406 Tuningthe Solr Server......................................................................................................................................406 SettingJavaSystemPropertiesforSolr..............................................................................................................................406 TuningtoCompleteDuringSetup......................................................................................................................................406 GeneralTuning...................................................................................................................................................................407 Other Resources.................................................................................................................................................................413 Tuning Apache Spark Applications...................................................................................................................413 TuningSparkShuffleOperations........................................................................................................................................413 ReducingtheSizeofDataStructures.................................................................................................................................419 Choosing Data Formats.....................................................................................................................................................420 Tuning YARN.....................................................................................................................................................420 Overview............................................................................................................................................................................420 Cluster Configuration.........................................................................................................................................................424 YARN Configuration...........................................................................................................................................................425 MapReduce Configuration.................................................................................................................................................426 Step7:MapReduceConfiguration.....................................................................................................................................426 Step7A:MapReduceSanityChecking................................................................................................................................427 ContinuousScheduling.......................................................................................................................................................427 ConfiguringYourClusterInClouderaManager.................................................................................................................428 Tuning JVM Garbage Collection.......................................................................................................................428 Resource Management........................................................................................431 Cloudera Manager Resource Management.....................................................................................................431 StaticServicePools...........................................................................................................................................432 LinuxControlGroups(cgroups)..........................................................................................................................................433 Dynamic Resource Pools..................................................................................................................................437 ManagingDynamicResourcePools...................................................................................................................................437 YARNPoolStatusandConfigurationOptions....................................................................................................................442 Defining ConfigurationSets...............................................................................................................................................444 Scheduling Configuration Sets...........................................................................................................................................445 AssigningApplicationsandQueriestoResourcePools......................................................................................................446 YARN(MRv2)andMapReduce(MRv1)Schedulers..........................................................................................449 ConfiguringtheFairScheduler...........................................................................................................................................450 EnablingandDisablingFairSchedulerPreemption...........................................................................................................453 DataStorageforMonitoringData....................................................................................................................454 ConfiguringServiceMonitorDataStorage........................................................................................................................454 ConfiguringHostMonitorDataStorage............................................................................................................................454 ViewingHostandServiceMonitorDataStorage...............................................................................................................454 DataGranularityandTime-SeriesMetricData..................................................................................................................455 MovingMonitoringDataonanActiveCluster...................................................................................................................455 HostMonitorandServiceMonitorMemoryConfiguration...............................................................................................455 DisablingMetricrollup......................................................................................................................................................457 Cluster Utilization Reports...............................................................................................................................457 ConfiguringtheClusterUtilizationReport.........................................................................................................................458 UsingtheClusterUtilizationReporttoManageResources...............................................................................................460 DownloadingClusterUtilizationReportsUsingtheClouderaManagerAPI......................................................................466 CreatingaCustomClusterUtilizationReport....................................................................................................................466 High Availability...................................................................................................478 HDFS High Availability......................................................................................................................................478 IntroductiontoHDFSHighAvailability...............................................................................................................................479 ConfiguringHardwareforHDFSHA...................................................................................................................................480 EnablingHDFSHA..............................................................................................................................................................481 DisablingandRedeployingHDFSHA..................................................................................................................................485 ConfiguringOtherCDHComponentstoUseHDFSHA.......................................................................................................485 AdministeringanHDFSHighAvailabilityCluster...............................................................................................................486 ChangingaNameserviceNameforHighlyAvailableHDFSUsingClouderaManager......................................................487 MapReduce(MRv1)andYARN(MRv2)HighAvailability..................................................................................488 YARN(MRv2)ResourceManagerHighAvailability.............................................................................................................488 WorkPreservingRecoveryforYARNComponents.............................................................................................................490 MapReduce(MRv1)JobTrackerHighAvailability..............................................................................................................492 ClouderaNavigatorKeyTrusteeServerHighAvailability.................................................................................493 ConfiguringKeyTrusteeServerHighAvailabilityUsingClouderaManager......................................................................493 RecoveringaKeyTrusteeServer........................................................................................................................................494 EnablingKeyTrusteeKMSHighAvailability.....................................................................................................494 EnablingNavigatorHSMKMSHighAvailability................................................................................................496 HSMKMSHighAvailabilityBackupandRecovery.............................................................................................................496 HighAvailabilityforOtherCDHComponents...................................................................................................497 HBaseHighAvailability......................................................................................................................................................497 OozieHighAvailability.......................................................................................................................................................502 Search High Availability.....................................................................................................................................................503 NavigatorDataManagementinaHighAvailabilityEnvironment....................................................................505 ConfiguringClouderaManagerforHighAvailabilityWithaLoadBalancer.....................................................506 IntroductiontoClouderaManagerDeploymentArchitecture...........................................................................................507 PrerequisitesforSettingupClouderaManagerHighAvailability......................................................................................508 ClouderaManagerFailoverProtection..............................................................................................................................509 High-LevelStepstoConfigureClouderaManagerHighAvailability.................................................................................510 DatabaseHighAvailabilityConfiguration..........................................................................................................................536 TLSandKerberosConfigurationforClouderaManagerHighAvailability.........................................................................536 Backup and Disaster Recovery..............................................................................540 PortRequirementsforBackupandDisasterRecovery.....................................................................................540 Data Replication...............................................................................................................................................541 ClouderaLicenseRequirementsforReplication.................................................................................................................541 SupportedandUnsupportedReplicationScenarios...........................................................................................................541 ReplicatingDirectorieswithThousandsofFilesandSubdirectories..................................................................................542 HDFSandHive/ImpalaReplicationToandFromCloudStorage........................................................................................542 SupportedReplicationScenariosforClustersusingIsilonStorage.....................................................................................543 BDR Log Retention.............................................................................................................................................................543 Designating a Replication Source......................................................................................................................................543 HDFS Replication................................................................................................................................................................545 Hive/ImpalaReplication.....................................................................................................................................................559 ReplicatingDatatoImpalaClusters...................................................................................................................................575 UsingSnapshotswithReplication......................................................................................................................................576 EnablingReplicationBetweenClusterswithKerberosAuthentication..............................................................................577 Replication of Encrypted Data...........................................................................................................................................580 HBase Replication..............................................................................................................................................................582 Snapshots.........................................................................................................................................................594 Cloudera Manager Snapshot Policies................................................................................................................................594 Managing HBase Snapshots..............................................................................................................................................598

Description:
or the applicable trademark holder. Hadoop and the Hadoop elephant logo are trademarks of the Apache Software. Foundation. All other trademarks, registered trademarks, product names and company .. Tuning Hive on Spark.
ebook img

Cloudera Administration PDF

551 Pages·2016·9.42 MB·English
by  
#cloudera #administration #apache spark
Save to my drive
Quick download
Download
Upgrade Premium
Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.

Download Cloudera Administration PDF Free - Full Version

by Unknow| 2016| 551 pages| 9.42| English

Download Cloudera Administration by in PDF format completely FREE. No registration required, no payment needed. Get instant access to this valuable resource on PDFdrive.to!

Free Download PDF

About Cloudera Administration

or the applicable trademark holder. Hadoop and the Hadoop elephant logo are trademarks of the Apache Software. Foundation. All other trademarks, registered trademarks, product names and company .. Tuning Hive on Spark.

Detailed Information

Author:Unknown
Publication Year:2016
Pages:551
Language:English
File Size:9.42
Format:PDF
Price:FREE
Download Free PDF

Safe & Secure Download - No registration required

Why Choose PDFdrive for Your Free Cloudera Administration Download?

  • 100% Free: No hidden fees or subscriptions required for one book every day.
  • No Registration: Immediate access is available without creating accounts for one book every day.
  • Safe and Secure: Clean downloads without malware or viruses
  • Multiple Formats: PDF, MOBI, Mpub,... optimized for all devices
  • Educational Resource: Supporting knowledge sharing and learning

Free Books Similar to Cloudera Administration

Frequently Asked Questions

Is it really free to download Cloudera Administration PDF?

Yes, on https://PDFdrive.to you can download Cloudera Administration by completely free. We don't require any payment, subscription, or registration to access this PDF file. For 3 books every day.

How can I read Cloudera Administration on my mobile device?

After downloading Cloudera Administration PDF, you can open it with any PDF reader app on your phone or tablet. We recommend using Adobe Acrobat Reader, Apple Books, or Google Play Books for the best reading experience.

Is this the full version of Cloudera Administration?

Yes, this is the complete PDF version of Cloudera Administration by Unknow. You will be able to read the entire content as in the printed version without missing any pages.

Is it legal to download Cloudera Administration PDF for free?

https://PDFdrive.to provides links to free educational resources available online. We do not store any files on our servers. Please be aware of copyright laws in your country before downloading.

The materials shared are intended for research, educational, and personal use in accordance with fair use principles.

Upgrade Premium
Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.
We use cookies

We use cookies to ensure you get the best browsing experience on our website. By clicking "Accept Cookies", you agree that we can store cookies on your device in accordance with our Terms and Privacy.