Jul 17 12:27:45.169134 ip-10-0-137-99 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 17 12:27:45.169142 ip-10-0-137-99 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 17 12:27:45.169148 ip-10-0-137-99 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 17 12:27:45.169511 ip-10-0-137-99 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 17 12:27:55.405795 ip-10-0-137-99 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 17 12:27:55.405814 ip-10-0-137-99 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot f2b5596f1e5f427c989da0f0dd12b91b --
Jul 17 12:29:50.311025 ip-10-0-137-99 systemd[1]: Starting Kubernetes Kubelet...
Jul 17 12:29:50.721526 ip-10-0-137-99 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 12:29:50.721526 ip-10-0-137-99 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 17 12:29:50.721526 ip-10-0-137-99 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 12:29:50.721526 ip-10-0-137-99 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 17 12:29:50.721526 ip-10-0-137-99 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 12:29:50.721526 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.684242    2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 17 12:29:50.721526 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688803    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 12:29:50.721526 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688819    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 12:29:50.721526 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688830    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 12:29:50.721526 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688835    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 12:29:50.721526 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688838    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 12:29:50.721526 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688841    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 12:29:50.721526 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688844    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 12:29:50.721526 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688847    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 12:29:50.723018 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688850    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 12:29:50.723018 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688853    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 12:29:50.723018 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688856    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 12:29:50.723018 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688859    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 12:29:50.723018 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688862    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 12:29:50.723018 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688865    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 12:29:50.723018 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688867    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 12:29:50.723018 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688870    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 12:29:50.723018 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688872    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 12:29:50.723018 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688877    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 12:29:50.723018 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688881    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 12:29:50.723018 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688884    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 12:29:50.723018 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688887    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 12:29:50.723018 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688891    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 12:29:50.723018 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688894    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 12:29:50.723018 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688897    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 12:29:50.723018 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688900    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 12:29:50.723018 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688902    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 12:29:50.723018 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688905    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 12:29:50.723018 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688908    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 12:29:50.723896 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688911    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 12:29:50.723896 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688914    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 12:29:50.723896 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688916    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 12:29:50.723896 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688918    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 12:29:50.723896 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688921    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 12:29:50.723896 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688924    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 12:29:50.723896 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688926    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 12:29:50.723896 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688937    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 12:29:50.723896 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688941    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 12:29:50.723896 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688943    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 12:29:50.723896 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688946    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 12:29:50.723896 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688948    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 12:29:50.723896 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688950    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 12:29:50.723896 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688953    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 12:29:50.723896 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688955    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 12:29:50.723896 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688958    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 12:29:50.723896 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688961    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 12:29:50.723896 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688964    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 12:29:50.723896 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688967    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 12:29:50.723896 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688970    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 12:29:50.724994 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688972    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 12:29:50.724994 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688975    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 12:29:50.724994 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688977    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 12:29:50.724994 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688980    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 12:29:50.724994 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688982    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 12:29:50.724994 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688984    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 12:29:50.724994 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688987    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 12:29:50.724994 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688989    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 12:29:50.724994 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688992    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 12:29:50.724994 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688995    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 12:29:50.724994 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.688997    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 12:29:50.724994 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689000    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 12:29:50.724994 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689002    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 12:29:50.724994 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689005    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 12:29:50.724994 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689007    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 12:29:50.724994 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689010    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 12:29:50.724994 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689012    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 12:29:50.724994 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689015    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 12:29:50.724994 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689017    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 12:29:50.724994 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689021    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 12:29:50.726062 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689029    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 12:29:50.726062 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689032    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 12:29:50.726062 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689035    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 12:29:50.726062 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689037    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 12:29:50.726062 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689040    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 12:29:50.726062 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689043    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 12:29:50.726062 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689047    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 12:29:50.726062 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689050    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 12:29:50.726062 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689052    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 12:29:50.726062 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689055    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 12:29:50.726062 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689061    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 12:29:50.726062 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689064    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 12:29:50.726062 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689068    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 12:29:50.726062 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689071    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 12:29:50.726062 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689074    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 12:29:50.726062 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689078    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 12:29:50.726062 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689080    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 12:29:50.726062 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689083    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 12:29:50.726062 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689085    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 12:29:50.726909 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689088    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 12:29:50.726909 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689090    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 12:29:50.726909 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689094    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 12:29:50.726909 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689096    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 12:29:50.726909 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689099    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 12:29:50.726909 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689102    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 12:29:50.726909 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689104    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 12:29:50.726909 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689107    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 12:29:50.726909 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689109    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 12:29:50.726909 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689113    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 12:29:50.726909 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689115    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 12:29:50.726909 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.689118    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 12:29:50.726909 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692627    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 12:29:50.726909 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692640    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 12:29:50.726909 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692643    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 12:29:50.726909 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692647    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 12:29:50.726909 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692650    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 12:29:50.726909 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692653    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 12:29:50.726909 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692657    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 12:29:50.726909 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692660    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 12:29:50.728019 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692663    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 12:29:50.728019 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692665    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 12:29:50.728019 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692671    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 12:29:50.728019 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692674    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 12:29:50.728019 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692677    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 12:29:50.728019 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692679    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 12:29:50.728019 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692682    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 12:29:50.728019 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692685    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 12:29:50.728019 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692688    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 12:29:50.728019 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692690    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 12:29:50.728019 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692693    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 12:29:50.728019 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692728    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 12:29:50.728019 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692732    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 12:29:50.728019 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692735    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 12:29:50.728019 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692738    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 12:29:50.728019 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692743    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 12:29:50.728019 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692746    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 12:29:50.728019 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692750    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 12:29:50.728019 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692753    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 12:29:50.728019 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692756    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 12:29:50.728937 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692758    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 12:29:50.728937 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692762    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 12:29:50.728937 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692765    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 12:29:50.728937 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692768    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 12:29:50.728937 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692771    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 12:29:50.728937 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692773    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 12:29:50.728937 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692776    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 12:29:50.728937 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692781    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 12:29:50.728937 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692784    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 12:29:50.728937 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692786    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 12:29:50.728937 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692790    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 12:29:50.728937 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692792    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 12:29:50.728937 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692798    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 12:29:50.728937 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692803    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 12:29:50.728937 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692807    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 12:29:50.728937 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692816    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 12:29:50.728937 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692819    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 12:29:50.728937 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692822    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 12:29:50.728937 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692825    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 12:29:50.730069 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692830    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 12:29:50.730069 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692833    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 12:29:50.730069 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692835    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 12:29:50.730069 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692839    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 12:29:50.730069 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692842    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 12:29:50.730069 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692845    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 12:29:50.730069 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692847    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 12:29:50.730069 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692850    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 12:29:50.730069 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692853    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 12:29:50.730069 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692856    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 12:29:50.730069 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692860    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 12:29:50.730069 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692864    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 12:29:50.730069 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692870    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 12:29:50.730069 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692873    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 12:29:50.730069 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692876    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 12:29:50.730069 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692878    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 12:29:50.730069 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692881    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 12:29:50.730069 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692883    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 12:29:50.730069 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692886    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 12:29:50.730939 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692888    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 12:29:50.730939 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692891    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 12:29:50.730939 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692894    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 12:29:50.730939 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692896    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 12:29:50.730939 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692899    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 12:29:50.730939 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692901    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 12:29:50.730939 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692909    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 12:29:50.730939 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692912    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 12:29:50.730939 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692914    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 12:29:50.730939 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692917    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 12:29:50.730939 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692920    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 12:29:50.730939 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692924    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 12:29:50.730939 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692927    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 12:29:50.730939 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692929    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 12:29:50.730939 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692932    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 12:29:50.730939 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692935    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 12:29:50.730939 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692937    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 12:29:50.730939 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692942    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 12:29:50.730939 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692945    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 12:29:50.730939 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692947    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 12:29:50.731814 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692950    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 12:29:50.731814 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692952    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 12:29:50.731814 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692955    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 12:29:50.731814 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692957    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 12:29:50.731814 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692960    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 12:29:50.731814 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692963    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 12:29:50.731814 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692966    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 12:29:50.731814 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692970    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 12:29:50.731814 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692975    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 12:29:50.731814 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692978    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 12:29:50.731814 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692983    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 12:29:50.731814 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692985    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 12:29:50.731814 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.692989    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 12:29:50.731814 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693619    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 12:29:50.731814 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693627    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 12:29:50.731814 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693630    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 12:29:50.731814 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693633    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 12:29:50.731814 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693638    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 12:29:50.731814 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693644    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 12:29:50.732739 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693648    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 12:29:50.732739 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693650    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 12:29:50.732739 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693654    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 12:29:50.732739 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693657    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 12:29:50.732739 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693659    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 12:29:50.732739 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693662    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 12:29:50.732739 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693665    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 12:29:50.732739 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693669    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 12:29:50.732739 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693675    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 12:29:50.732739 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693678    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 12:29:50.732739 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693681    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 12:29:50.732739 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693684    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 12:29:50.732739 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693687    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 12:29:50.732739 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693690    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 12:29:50.732739 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693692    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 12:29:50.732739 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693713    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 12:29:50.732739 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693718    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 12:29:50.732739 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693721    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 12:29:50.732739 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693725    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 12:29:50.733587 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693729    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 12:29:50.733587 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693735    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 12:29:50.733587 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693739    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 12:29:50.733587 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693742    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 12:29:50.733587 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693744    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 12:29:50.733587 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693747    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 12:29:50.733587 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693750    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 12:29:50.733587 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693753    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 12:29:50.733587 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693756    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 12:29:50.733587 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693759    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 12:29:50.733587 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693761    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 12:29:50.733587 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693764    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 12:29:50.733587 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693768    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 12:29:50.733587 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693771    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 12:29:50.733587 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693776    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 12:29:50.733587 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693779    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 12:29:50.733587 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693782    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 12:29:50.733587 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693785    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 12:29:50.733587 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693788    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 12:29:50.733587 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693791    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 12:29:50.734444 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693794    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 12:29:50.734444 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693796    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 12:29:50.734444 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693799    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 12:29:50.734444 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693802    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 12:29:50.734444 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693805    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 12:29:50.734444 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693810    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 12:29:50.734444 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693813    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 12:29:50.734444 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693818    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 12:29:50.734444 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693821    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 12:29:50.734444 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693824    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 12:29:50.734444 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693826    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 12:29:50.734444 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693829    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 12:29:50.734444 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693832    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 12:29:50.734444 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693834    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 12:29:50.734444 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693837    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 12:29:50.734444 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693839    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 12:29:50.734444 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693842    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 12:29:50.734444 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693844    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 12:29:50.734444 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693849    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 12:29:50.734444 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693853    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 12:29:50.735340 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693856    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 12:29:50.735340 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693858    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 12:29:50.735340 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693861    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 12:29:50.735340 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693863    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 12:29:50.735340 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693867    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 12:29:50.735340 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693870    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 12:29:50.735340 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693872    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 12:29:50.735340 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693875    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 12:29:50.735340 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693878    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 12:29:50.735340 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693881    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 12:29:50.735340 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693885    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 12:29:50.735340 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693888    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 12:29:50.735340 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693891    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 12:29:50.735340 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693894    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 12:29:50.735340 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693896    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 12:29:50.735340 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693899    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 12:29:50.735340 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693902    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 12:29:50.735340 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693904    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 12:29:50.735340 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693907    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 12:29:50.735340 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693911    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 12:29:50.736236 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693913    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 12:29:50.736236 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693916    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 12:29:50.736236 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693921    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 12:29:50.736236 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693923    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 12:29:50.736236 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693926    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 12:29:50.736236 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693929    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 12:29:50.736236 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693931    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 12:29:50.736236 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693934    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 12:29:50.736236 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693937    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 12:29:50.736236 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693939    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 12:29:50.736236 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693942    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 12:29:50.736236 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693944    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 12:29:50.736236 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693947    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 12:29:50.736236 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.693950    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 12:29:50.736236 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694039    2564 flags.go:64] FLAG: --address="0.0.0.0"
Jul 17 12:29:50.736236 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694047    2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 17 12:29:50.736236 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694055    2564 flags.go:64] FLAG: --anonymous-auth="true"
Jul 17 12:29:50.736236 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694059    2564 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 17 12:29:50.736236 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694067    2564 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 17 12:29:50.736236 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694070    2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 17 12:29:50.736236 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694074    2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 17 12:29:50.740597 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694079    2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 17 12:29:50.740597 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694083    2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 17 12:29:50.740597 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694086    2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 17 12:29:50.740597 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694089    2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 17 12:29:50.740597 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694093    2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 17 12:29:50.740597 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694105    2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 17 12:29:50.740597 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694109    2564 flags.go:64] FLAG: --cgroup-root=""
Jul 17 12:29:50.740597 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694112    2564 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 17 12:29:50.740597 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694115    2564 flags.go:64] FLAG: --client-ca-file=""
Jul 17 12:29:50.740597 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694118    2564 flags.go:64] FLAG: --cloud-provider="external"
Jul 17 12:29:50.740597 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694123    2564 flags.go:64] FLAG: --cluster-dns="[]"
Jul 17 12:29:50.740597 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694127    2564 flags.go:64] FLAG: --cluster-domain=""
Jul 17 12:29:50.740597 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694130    2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 17 12:29:50.740597 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694135    2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 17 12:29:50.740597 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694139    2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 17 12:29:50.740597 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694143    2564 flags.go:64] FLAG: --container-log-max-files="5"
Jul 17 12:29:50.740597 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694147    2564 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 17 12:29:50.740597 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694152    2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 17 12:29:50.740597 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694155    2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 17 12:29:50.740597 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694159    2564 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 17 12:29:50.740597 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694162    2564 flags.go:64] FLAG: --contention-profiling="false"
Jul 17 12:29:50.740597 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694167    2564 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 17 12:29:50.740597 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694170    2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 17 12:29:50.740597 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694173    2564 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 17 12:29:50.741739 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694177    2564 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 17 12:29:50.741739 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694181    2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 17 12:29:50.741739 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694184    2564 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 17 12:29:50.741739 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694189    2564 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 17 12:29:50.741739 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694192    2564 flags.go:64] FLAG: --enable-load-reader="false"
Jul 17 12:29:50.741739 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694195    2564 flags.go:64] FLAG: --enable-server="true"
Jul 17 12:29:50.741739 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694200    2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 17 12:29:50.741739 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694205    2564 flags.go:64] FLAG: --event-burst="100"
Jul 17 12:29:50.741739 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694208    2564 flags.go:64] FLAG: --event-qps="50"
Jul 17 12:29:50.741739 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694211    2564 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 17 12:29:50.741739 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694215    2564 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 17 12:29:50.741739 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694218    2564 flags.go:64] FLAG: --eviction-hard=""
Jul 17 12:29:50.741739 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694222    2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 17 12:29:50.741739 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694225    2564 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 17 12:29:50.741739 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694231    2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 17 12:29:50.741739 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694234    2564 flags.go:64] FLAG: --eviction-soft=""
Jul 17 12:29:50.741739 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694237    2564 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 17 12:29:50.741739 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694241    2564 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 17 12:29:50.741739 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694248    2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 17 12:29:50.741739 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694251    2564 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 17 12:29:50.741739 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694255    2564 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 17 12:29:50.741739 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694258    2564 flags.go:64] FLAG: --fail-swap-on="true"
Jul 17 12:29:50.741739 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694269    2564 flags.go:64] FLAG: --feature-gates=""
Jul 17 12:29:50.741739 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694273    2564 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 17 12:29:50.741739 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694276    2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 17 12:29:50.749621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694279    2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 17 12:29:50.749621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694283    2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 17 12:29:50.749621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694286    2564 flags.go:64] FLAG: --healthz-port="10248"
Jul 17 12:29:50.749621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694289    2564 flags.go:64] FLAG: --help="false"
Jul 17 12:29:50.749621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694292    2564 flags.go:64] FLAG: --hostname-override="ip-10-0-137-99.ec2.internal"
Jul 17 12:29:50.749621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694295    2564 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 17 12:29:50.749621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694300    2564 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 17 12:29:50.749621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694303    2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 17 12:29:50.749621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694307    2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 17 12:29:50.749621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694310    2564 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 17 12:29:50.749621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694314    2564 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 17 12:29:50.749621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694317    2564 flags.go:64] FLAG: --image-service-endpoint=""
Jul 17 12:29:50.749621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694320    2564 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 17 12:29:50.749621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694323    2564 flags.go:64] FLAG: --kube-api-burst="100"
Jul 17 12:29:50.749621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694326    2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 17 12:29:50.749621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694331    2564 flags.go:64] FLAG: --kube-api-qps="50"
Jul 17 12:29:50.749621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694334    2564 flags.go:64] FLAG: --kube-reserved=""
Jul 17 12:29:50.749621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694337    2564 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 17 12:29:50.749621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694340    2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 17 12:29:50.749621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694343    2564 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 17 12:29:50.749621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694346    2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 17 12:29:50.749621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694349    2564 flags.go:64] FLAG: --lock-file=""
Jul 17 12:29:50.749621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694352    2564 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 17 12:29:50.749621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694357    2564 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 17 12:29:50.750742 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694360    2564 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 17 12:29:50.750742 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694366    2564 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 17 12:29:50.750742 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694369    2564 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 17 12:29:50.750742 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694372    2564 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 17 12:29:50.750742 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694375    2564 flags.go:64] FLAG: --logging-format="text"
Jul 17 12:29:50.750742 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694378    2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 17 12:29:50.750742 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694381    2564 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 17 12:29:50.750742 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694384    2564 flags.go:64] FLAG: --manifest-url=""
Jul 17 12:29:50.750742 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694389    2564 flags.go:64] FLAG: --manifest-url-header=""
Jul 17 12:29:50.750742 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694394    2564 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 17 12:29:50.750742 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694398    2564 flags.go:64] FLAG: --max-open-files="1000000"
Jul 17 12:29:50.750742 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694402    2564 flags.go:64] FLAG: --max-pods="110"
Jul 17 12:29:50.750742 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694405    2564 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 17 12:29:50.750742 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694408    2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 17 12:29:50.750742 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694411    2564 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 17 12:29:50.750742 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694414    2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 17 12:29:50.750742 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694419    2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 17 12:29:50.750742 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694422    2564 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 17 12:29:50.750742 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694426    2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 17 12:29:50.750742 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694434    2564 flags.go:64] FLAG: --node-status-max-images="50"
Jul 17 12:29:50.750742 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694437    2564 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 17 12:29:50.750742 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694440    2564 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 17 12:29:50.750742 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694444    2564 flags.go:64] FLAG: --pod-cidr=""
Jul 17 12:29:50.752065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694449    2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 17 12:29:50.752065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694454    2564 flags.go:64] FLAG: --pod-manifest-path=""
Jul 17 12:29:50.752065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694457    2564 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 17 12:29:50.752065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694461    2564 flags.go:64] FLAG: --pods-per-core="0"
Jul 17 12:29:50.752065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694464    2564 flags.go:64] FLAG: --port="10250"
Jul 17 12:29:50.752065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694467    2564 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 17 12:29:50.752065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694470    2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b18499debb2c3eea"
Jul 17 12:29:50.752065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694473    2564 flags.go:64] FLAG: --qos-reserved=""
Jul 17 12:29:50.752065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694478    2564 flags.go:64] FLAG: --read-only-port="10255"
Jul 17 12:29:50.752065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694481    2564 flags.go:64] FLAG: --register-node="true"
Jul 17 12:29:50.752065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694486    2564 flags.go:64] FLAG: --register-with-taints=""
Jul 17 12:29:50.752065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694490    2564 flags.go:64] FLAG: --registry-burst="10"
Jul 17 12:29:50.752065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694493    2564 flags.go:64] FLAG: --registry-qps="5"
Jul 17 12:29:50.752065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694496    2564 flags.go:64] FLAG: --reserved-cpus=""
Jul 17 12:29:50.752065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694499    2564 flags.go:64] FLAG: --reserved-memory=""
Jul 17 12:29:50.752065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694503    2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 17 12:29:50.752065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694506    2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 17 12:29:50.752065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694511    2564 flags.go:64] FLAG: --rotate-certificates="false"
Jul 17 12:29:50.752065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694514    2564 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 17 12:29:50.752065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694517    2564 flags.go:64] FLAG: --runonce="false"
Jul 17 12:29:50.752065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694521    2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 17 12:29:50.752065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694524    2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 17 12:29:50.752065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694527    2564 flags.go:64] FLAG: --seccomp-default="false"
Jul 17 12:29:50.752065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694530    2564 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 17 12:29:50.752065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694533    2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 17 12:29:50.753168 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694538    2564 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 17 12:29:50.753168 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694541    2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 17 12:29:50.753168 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694544    2564 flags.go:64] FLAG: --storage-driver-password="root"
Jul 17 12:29:50.753168 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694548    2564 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 17 12:29:50.753168 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694552    2564 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 17 12:29:50.753168 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694555    2564 flags.go:64] FLAG: --storage-driver-user="root"
Jul 17 12:29:50.753168 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694558    2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 17 12:29:50.753168 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694561    2564 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 17 12:29:50.753168 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694564    2564 flags.go:64] FLAG: --system-cgroups=""
Jul 17 12:29:50.753168 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694569    2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 17 12:29:50.753168 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694574    2564 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 17 12:29:50.753168 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694577    2564 flags.go:64] FLAG: --tls-cert-file=""
Jul 17 12:29:50.753168 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694580    2564 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 17 12:29:50.753168 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694585    2564 flags.go:64] FLAG: --tls-min-version=""
Jul 17 12:29:50.753168 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694591    2564 flags.go:64] FLAG: --tls-private-key-file=""
Jul 17 12:29:50.753168 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694594    2564 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 17 12:29:50.753168 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694600    2564 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 17 12:29:50.753168 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694604    2564 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 17 12:29:50.753168 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694608    2564 flags.go:64] FLAG: --v="2"
Jul 17 12:29:50.753168 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694612    2564 flags.go:64] FLAG: --version="false"
Jul 17 12:29:50.753168 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694616    2564 flags.go:64] FLAG: --vmodule=""
Jul 17 12:29:50.753168 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694619    2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 17 12:29:50.753168 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.694623    2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 17 12:29:50.753168 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694846    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 12:29:50.753168 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694851    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 12:29:50.754262 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694854    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 12:29:50.754262 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694857    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 12:29:50.754262 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694863    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 12:29:50.754262 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694867    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 12:29:50.754262 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694870    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 12:29:50.754262 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694873    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 12:29:50.754262 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694876    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 12:29:50.754262 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694878    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 12:29:50.754262 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694881    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 12:29:50.754262 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694883    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 12:29:50.754262 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694888    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 12:29:50.754262 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694890    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 12:29:50.754262 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694893    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 12:29:50.754262 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694895    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 12:29:50.754262 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694900    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 12:29:50.754262 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694903    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 12:29:50.754262 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694905    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 12:29:50.754262 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694908    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 12:29:50.754262 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694911    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 12:29:50.754262 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694913    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 12:29:50.826470 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694916    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 12:29:50.826470 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694919    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 12:29:50.826470 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694921    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 12:29:50.826470 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694924    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 12:29:50.826470 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694928    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 12:29:50.826470 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694931    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 12:29:50.826470 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694934    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 12:29:50.826470 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694939    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 12:29:50.826470 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694941    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 12:29:50.826470 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694944    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 12:29:50.826470 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694947    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 12:29:50.826470 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694949    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 12:29:50.826470 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694951    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 12:29:50.826470 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694954    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 12:29:50.826470 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694957    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 12:29:50.826470 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694960    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 12:29:50.826470 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694963    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 12:29:50.826470 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694965    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 12:29:50.826470 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694968    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 12:29:50.826470 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694973    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 12:29:50.756492 ip-10-0-137-99 systemd[1]: Started Kubernetes Kubelet.
Jul 17 12:29:50.827271 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694976    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 12:29:50.827271 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694978    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 12:29:50.827271 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694982    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 12:29:50.827271 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694985    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 12:29:50.827271 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694988    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 12:29:50.827271 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694990    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 12:29:50.827271 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694993    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 12:29:50.827271 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.694997    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 12:29:50.827271 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695000    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 12:29:50.827271 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695003    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 12:29:50.827271 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695005    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 12:29:50.827271 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695008    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 12:29:50.827271 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695013    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 12:29:50.827271 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695016    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 12:29:50.827271 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695019    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 12:29:50.827271 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695021    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 12:29:50.827271 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695025    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 12:29:50.827271 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695028    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 12:29:50.827271 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695030    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 12:29:50.827271 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695033    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 12:29:50.828165 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695035    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 12:29:50.828165 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695039    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 12:29:50.828165 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695043    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 12:29:50.828165 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695046    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 12:29:50.828165 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695051    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 12:29:50.828165 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695053    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 12:29:50.828165 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695056    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 12:29:50.828165 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695059    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 12:29:50.828165 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695061    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 12:29:50.828165 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695064    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 12:29:50.828165 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695067    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 12:29:50.828165 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695070    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 12:29:50.828165 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695072    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 12:29:50.828165 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695075    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 12:29:50.828165 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695078    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 12:29:50.828165 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695081    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 12:29:50.828165 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695086    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 12:29:50.828165 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695089    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 12:29:50.828165 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695091    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 12:29:50.828790 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695094    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 12:29:50.828790 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695096    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 12:29:50.828790 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695099    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 12:29:50.828790 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695101    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 12:29:50.828790 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695104    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 12:29:50.828790 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695106    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 12:29:50.828790 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695109    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 12:29:50.828790 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695112    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 12:29:50.828790 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695115    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 12:29:50.828790 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695118    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 12:29:50.828790 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695123    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 12:29:50.828790 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695126    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 12:29:50.828790 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695129    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 12:29:50.828790 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695131    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 12:29:50.828790 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695135    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 12:29:50.828790 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695138    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 12:29:50.828790 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695141    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 12:29:50.828790 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.695144    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 12:29:50.829366 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.695150    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 12:29:50.829366 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.702614    2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 17 12:29:50.829366 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.702636    2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 17 12:29:50.829366 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.703335    2564 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 17 12:29:50.829366 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.703345    2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 17 12:29:50.829366 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703413    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 12:29:50.829366 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703419    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 12:29:50.829366 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703422    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 12:29:50.829366 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703426    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 12:29:50.829366 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703429    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 12:29:50.829366 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703431    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 12:29:50.829366 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703434    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 12:29:50.829366 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703438    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 12:29:50.829366 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703441    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 12:29:50.829366 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703443    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 12:29:50.829862 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703449    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 12:29:50.829862 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703452    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 12:29:50.829862 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703455    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 12:29:50.829862 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703458    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 12:29:50.829862 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703460    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 12:29:50.829862 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703463    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 12:29:50.829862 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703466    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 12:29:50.829862 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703469    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 12:29:50.829862 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703472    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 12:29:50.829862 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703476    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 12:29:50.829862 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703481    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 12:29:50.829862 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703484    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 12:29:50.829862 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703486    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 12:29:50.829862 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703489    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 12:29:50.829862 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703494    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 12:29:50.829862 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703497    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 12:29:50.829862 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703501    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 12:29:50.829862 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703505    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 12:29:50.829862 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703508    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 12:29:50.830484 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703511    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 12:29:50.830484 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703515    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 12:29:50.830484 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703517    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 12:29:50.830484 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703520    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 12:29:50.830484 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703523    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 12:29:50.830484 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703526    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 12:29:50.830484 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703529    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 12:29:50.830484 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703532    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 12:29:50.830484 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703535    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 12:29:50.830484 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703537    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 12:29:50.830484 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703540    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 12:29:50.830484 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703543    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 12:29:50.830484 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703547    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 12:29:50.830484 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703549    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 12:29:50.830484 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703552    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 12:29:50.830484 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703554    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 12:29:50.830484 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703557    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 12:29:50.830484 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703559    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 12:29:50.830484 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703562    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 12:29:50.830484 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703565    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 12:29:50.831111 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703569    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 12:29:50.831111 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703572    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 12:29:50.831111 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703575    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 12:29:50.831111 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703577    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 12:29:50.831111 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703580    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 12:29:50.831111 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703582    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 12:29:50.831111 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703585    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 12:29:50.831111 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703589    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 12:29:50.831111 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703593    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 12:29:50.831111 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703596    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 12:29:50.831111 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703599    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 12:29:50.831111 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703602    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 12:29:50.831111 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703604    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 12:29:50.831111 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703607    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 12:29:50.831111 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703609    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 12:29:50.831111 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703612    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 12:29:50.831111 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703615    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 12:29:50.831111 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703617    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 12:29:50.831111 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703621    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 12:29:50.831711 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703624    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 12:29:50.831711 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703626    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 12:29:50.831711 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703629    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 12:29:50.831711 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703632    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 12:29:50.831711 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703635    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 12:29:50.831711 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703637    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 12:29:50.831711 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703640    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 12:29:50.831711 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703642    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 12:29:50.831711 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703645    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 12:29:50.831711 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703647    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 12:29:50.831711 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703650    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 12:29:50.831711 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703652    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 12:29:50.831711 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703655    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 12:29:50.831711 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703658    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 12:29:50.831711 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703660    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 12:29:50.831711 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703663    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 12:29:50.831711 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703665    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 12:29:50.831711 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703668    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 12:29:50.831711 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703671    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 12:29:50.831711 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703674    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 12:29:50.832610 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703681    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 12:29:50.832610 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703684    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 12:29:50.832610 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703688    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 12:29:50.832610 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703691    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 12:29:50.832610 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703693    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 12:29:50.832610 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703711    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 12:29:50.832610 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703714    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 12:29:50.832610 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703717    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 12:29:50.832610 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703719    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 12:29:50.832610 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703722    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 12:29:50.832610 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703725    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 12:29:50.832610 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.703730    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 12:29:50.832610 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703864    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 12:29:50.832610 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703869    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 12:29:50.832610 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703872    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 12:29:50.833501 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703876    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 12:29:50.833501 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703879    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 12:29:50.833501 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703883    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 12:29:50.833501 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703887    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 12:29:50.833501 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703890    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 12:29:50.833501 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703893    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 12:29:50.833501 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703895    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 12:29:50.833501 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703898    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 12:29:50.833501 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703901    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 12:29:50.833501 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703904    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 12:29:50.833501 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703907    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 12:29:50.833501 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703910    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 12:29:50.833501 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703913    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 12:29:50.833501 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703915    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 12:29:50.833501 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703918    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 12:29:50.833501 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703920    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 12:29:50.833501 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703923    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 12:29:50.833501 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703928    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 12:29:50.833501 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703931    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 12:29:50.834343 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703934    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 12:29:50.834343 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703937    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 12:29:50.834343 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703940    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 12:29:50.834343 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703943    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 12:29:50.834343 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703946    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 12:29:50.834343 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703948    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 12:29:50.834343 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703951    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 12:29:50.834343 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703953    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 12:29:50.834343 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703956    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 12:29:50.834343 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703959    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 12:29:50.834343 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703961    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 12:29:50.834343 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703964    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 12:29:50.834343 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703966    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 12:29:50.834343 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703970    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 12:29:50.834343 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703980    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 12:29:50.834343 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703983    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 12:29:50.834343 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703986    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 12:29:50.834343 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703990    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 12:29:50.834343 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703993    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 12:29:50.834343 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703996    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 12:29:50.835410 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.703999    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 12:29:50.835410 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704001    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 12:29:50.835410 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704004    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 12:29:50.835410 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704007    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 12:29:50.835410 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704010    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 12:29:50.835410 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704013    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 12:29:50.835410 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704016    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 12:29:50.835410 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704018    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 12:29:50.835410 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704021    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 12:29:50.835410 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704024    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 12:29:50.835410 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704027    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 12:29:50.835410 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704030    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 12:29:50.835410 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704033    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 12:29:50.835410 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704036    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 12:29:50.835410 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704040    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 12:29:50.835410 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704043    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 12:29:50.835410 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704045    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 12:29:50.835410 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704048    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 12:29:50.835410 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704050    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 12:29:50.836296 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704053    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 12:29:50.836296 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704056    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 12:29:50.836296 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704058    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 12:29:50.836296 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704061    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 12:29:50.836296 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704063    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 12:29:50.836296 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704066    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 12:29:50.836296 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704069    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 12:29:50.836296 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704072    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 12:29:50.836296 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704074    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 12:29:50.836296 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704077    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 12:29:50.836296 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704080    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 12:29:50.836296 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704082    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 12:29:50.836296 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704084    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 12:29:50.836296 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704087    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 12:29:50.836296 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704090    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 12:29:50.836296 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704092    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 12:29:50.836296 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704095    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 12:29:50.836296 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704098    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 12:29:50.836296 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704100    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 12:29:50.836296 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704103    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 12:29:50.837394 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704105    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 12:29:50.837394 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704108    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 12:29:50.837394 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704110    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 12:29:50.837394 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704114    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 12:29:50.837394 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704117    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 12:29:50.837394 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704119    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 12:29:50.837394 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704122    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 12:29:50.837394 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704124    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 12:29:50.837394 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704127    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 12:29:50.837394 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704130    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 12:29:50.837394 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704133    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 12:29:50.837394 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704135    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 12:29:50.837394 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704138    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 12:29:50.837394 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704140    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 12:29:50.837394 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704143    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 12:29:50.837394 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704145    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 12:29:50.837394 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704148    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 12:29:50.837394 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:50.704150    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 12:29:50.840596 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.704156    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 12:29:50.840596 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.704293    2564 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 17 12:29:50.840596 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.706485    2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 17 12:29:50.840596 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.707722    2564 server.go:1019] "Starting client certificate rotation"
Jul 17 12:29:50.840596 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.707822    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 12:29:50.840596 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.707864    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 12:29:50.840596 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.709974    2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 12:29:50.840596 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.711185    2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 12:29:50.840596 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.724495    2564 log.go:25] "Validated CRI v1 runtime API"
Jul 17 12:29:50.840596 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.726888    2564 log.go:25] "Validated CRI v1 image API"
Jul 17 12:29:50.840596 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.727684    2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 17 12:29:50.840596 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.731148    2564 fs.go:135] Filesystem UUIDs: map[3e8f5d3e-98fe-4fe8-a7c6-49489abcca08:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ceb86dd1-fe2d-4c07-a569-cbfb350c8346:/dev/nvme0n1p4]
Jul 17 12:29:50.841256 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.731170    2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 17 12:29:50.841256 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.734812    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 12:29:50.959441 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.737649    2564 manager.go:216] Machine: {Timestamp:2026-07-17 12:29:50.737115218 +0000 UTC m=+0.317227229 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199983 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2648c95c823a580a097354c8e68c86 SystemUUID:ec2648c9-5c82-3a58-0a09-7354c8e68c86 BootID:f2b5596f-1e5f-427c-989d-a0f0dd12b91b Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1b:5f:a6:c6:19 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1b:5f:a6:c6:19 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:bf:e3:42:09:68 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 17 12:29:50.959441 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.738449    2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 17 12:29:50.959441 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.738532    2564 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 17 12:29:50.959441 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.741087    2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 17 12:29:50.959441 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.741115    2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-99.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 17 12:29:50.959441 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.741824    2564 topology_manager.go:138] "Creating topology manager with none policy"
Jul 17 12:29:50.959441 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.741832    2564 container_manager_linux.go:309] "Creating device plugin manager"
Jul 17 12:29:50.959441 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.741840    2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 12:29:50.959441 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.741851    2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 12:29:50.959441 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.743208    2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 17 12:29:50.959441 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.747171    2564 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 12:29:50.959441 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.747294    2564 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 17 12:29:50.959441 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.747321    2564 kubelet.go:486] "Attempting to sync node with API server"
Jul 17 12:29:50.959441 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.747336    2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 17 12:29:50.959441 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.747346    2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 17 12:29:50.959441 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.747354    2564 kubelet.go:398] "Adding apiserver pod source"
Jul 17 12:29:50.959441 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.747363    2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 17 12:29:50.959441 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.747764    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.747783    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.749802    2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.751204    2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.751231    2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.751850    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.751863    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.751869    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.751876    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.751885    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.751890    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.751898    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.751903    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.751910    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.751915    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.751923    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.751928    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.752539    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.752546    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.755607    2564 server.go:1268] "Started kubelet"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.755683    2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.755730    2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.755823    2564 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.757040    2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.757225    2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:50.757793    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:50.757794    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.758214    2564 server.go:310] "Adding debug handlers to kubelet server"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:50.761595    2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:50.763206    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-99.ec2.internal.18c31390f9c61dc3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-99.ec2.internal,UID:ip-10-0-137-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-99.ec2.internal,},FirstTimestamp:2026-07-17 12:29:50.755577283 +0000 UTC m=+0.335689294,LastTimestamp:2026-07-17 12:29:50.755577283 +0000 UTC m=+0.335689294,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-99.ec2.internal,}"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.764175    2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.764184    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:50.764235    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-99.ec2.internal\" not found"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.764728    2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.764730    2564 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.764756    2564 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.764820    2564 reconstruct.go:97] "Volume reconstruction finished"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.764830    2564 reconciler.go:29] "Reconciler: start to sync state"
Jul 17 12:29:50.960694 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:50.767530    2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:50.767566    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.770948    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.771523    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n66v5"
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.778866    2564 factory.go:55] Registering systemd factory
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.778884    2564 factory.go:223] Registration of the systemd container factory successfully
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.779141    2564 factory.go:153] Registering CRI-O factory
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.779153    2564 factory.go:223] Registration of the crio container factory successfully
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.779213    2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.779240    2564 factory.go:103] Registering Raw factory
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.779257    2564 manager.go:1196] Started watching for new ooms in manager
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.779618    2564 manager.go:349] Starting recovery of all containers
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.780672    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n66v5"
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.798270    2564 manager.go:354] Recovery completed
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.802497    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.804443    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-99.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.804480    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-99.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.804496    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-99.ec2.internal" event="NodeHasSufficientPID"
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.805020    2564 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.805033    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.805033    2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.805096    2564 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:50.806628    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-99.ec2.internal.18c31390fcb00431  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-99.ec2.internal,UID:ip-10-0-137-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-99.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-99.ec2.internal,},FirstTimestamp:2026-07-17 12:29:50.804460593 +0000 UTC m=+0.384572604,LastTimestamp:2026-07-17 12:29:50.804460593 +0000 UTC m=+0.384572604,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-99.ec2.internal,}"
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.807334    2564 policy_none.go:49] "None policy: Start"
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.807347    2564 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.807356    2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.809417    2564 policy_none.go:47] "Start"
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.809963    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-99.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.809987    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-99.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.809998    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-99.ec2.internal" event="NodeHasSufficientPID"
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:50.866838    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-99.ec2.internal\" not found"
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.893916    2564 manager.go:337] "Starting Device Plugin manager"
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:50.893973    2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.893985    2564 server.go:87] "Starting device plugin registration server"
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.894297    2564 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.894311    2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.895460    2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.895746    2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.895754    2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:50.895985    2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 17 12:29:50.961898 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:50.896043    2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-99.ec2.internal\" not found"
Jul 17 12:29:51.061104 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:50.981027    2564 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-99.ec2.internal\" not found" node="ip-10-0-137-99.ec2.internal"
Jul 17 12:29:51.061104 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.995123    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 12:29:51.061104 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.996122    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-99.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 12:29:51.061104 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.996145    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-99.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 12:29:51.061104 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.996157    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-99.ec2.internal" event="NodeHasSufficientPID"
Jul 17 12:29:51.061104 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:50.996194    2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-99.ec2.internal"
Jul 17 12:29:51.061104 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.005682    2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-99.ec2.internal"
Jul 17 12:29:51.061104 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:51.005705    2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-99.ec2.internal\": node \"ip-10-0-137-99.ec2.internal\" not found"
Jul 17 12:29:51.061104 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.008214    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 17 12:29:51.061104 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.008249    2564 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 17 12:29:51.061104 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.008270    2564 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 17 12:29:51.061104 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:51.008309    2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 17 12:29:51.061104 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.010542    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 12:29:51.061104 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:51.059971    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-99.ec2.internal\" not found"
Jul 17 12:29:51.177588 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.110781    2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-99.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-99.ec2.internal"]
Jul 17 12:29:51.177588 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.110874    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 12:29:51.177588 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.112577    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-99.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 12:29:51.177588 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.112604    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-99.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 12:29:51.177588 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.112613    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-99.ec2.internal" event="NodeHasSufficientPID"
Jul 17 12:29:51.177588 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.115282    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 12:29:51.177588 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.115455    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-99.ec2.internal"
Jul 17 12:29:51.177588 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.115487    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 12:29:51.177588 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.116134    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-99.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 12:29:51.177588 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.116154    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-99.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 12:29:51.177588 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.116165    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-99.ec2.internal" event="NodeHasSufficientPID"
Jul 17 12:29:51.177588 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.116427    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-99.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 12:29:51.177588 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.116442    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-99.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 12:29:51.177588 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.116451    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-99.ec2.internal" event="NodeHasSufficientPID"
Jul 17 12:29:51.177588 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.118931    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-99.ec2.internal"
Jul 17 12:29:51.177588 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.118951    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 12:29:51.177588 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.119850    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-99.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 12:29:51.177588 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.119866    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-99.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 12:29:51.177588 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.119875    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-99.ec2.internal" event="NodeHasSufficientPID"
Jul 17 12:29:51.823821 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:51.752157    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-99.ec2.internal\" not found"
Jul 17 12:29:51.823821 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.752180    2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 17 12:29:51.823821 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.752346    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 17 12:29:51.823821 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.771030    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c8cbb35a0cc4730bb524456da0fe6cdf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-99.ec2.internal\" (UID: \"c8cbb35a0cc4730bb524456da0fe6cdf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-99.ec2.internal"
Jul 17 12:29:51.823821 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.771059    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8cbb35a0cc4730bb524456da0fe6cdf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-99.ec2.internal\" (UID: \"c8cbb35a0cc4730bb524456da0fe6cdf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-99.ec2.internal"
Jul 17 12:29:51.823821 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.771078    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f283087a6d8d1084ff25d034c2bbc6d1-config\") pod \"kube-apiserver-proxy-ip-10-0-137-99.ec2.internal\" (UID: \"f283087a6d8d1084ff25d034c2bbc6d1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-99.ec2.internal"
Jul 17 12:29:51.823821 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.787022    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 12:24:51 +0000 UTC" deadline="2028-01-03 12:58:42.451223311 +0000 UTC"
Jul 17 12:29:51.823821 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.787054    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12840h28m50.664173291s"
Jul 17 12:29:51.949827 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:51.853241    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-99.ec2.internal\" not found"
Jul 17 12:29:51.949827 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.871854    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c8cbb35a0cc4730bb524456da0fe6cdf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-99.ec2.internal\" (UID: \"c8cbb35a0cc4730bb524456da0fe6cdf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-99.ec2.internal"
Jul 17 12:29:51.949827 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.871896    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8cbb35a0cc4730bb524456da0fe6cdf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-99.ec2.internal\" (UID: \"c8cbb35a0cc4730bb524456da0fe6cdf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-99.ec2.internal"
Jul 17 12:29:51.949827 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.871913    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f283087a6d8d1084ff25d034c2bbc6d1-config\") pod \"kube-apiserver-proxy-ip-10-0-137-99.ec2.internal\" (UID: \"f283087a6d8d1084ff25d034c2bbc6d1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-99.ec2.internal"
Jul 17 12:29:51.949827 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.871947    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f283087a6d8d1084ff25d034c2bbc6d1-config\") pod \"kube-apiserver-proxy-ip-10-0-137-99.ec2.internal\" (UID: \"f283087a6d8d1084ff25d034c2bbc6d1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-99.ec2.internal"
Jul 17 12:29:51.949827 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.871961    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c8cbb35a0cc4730bb524456da0fe6cdf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-99.ec2.internal\" (UID: \"c8cbb35a0cc4730bb524456da0fe6cdf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-99.ec2.internal"
Jul 17 12:29:51.949827 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.871968    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8cbb35a0cc4730bb524456da0fe6cdf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-99.ec2.internal\" (UID: \"c8cbb35a0cc4730bb524456da0fe6cdf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-99.ec2.internal"
Jul 17 12:29:51.949827 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:51.915820    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 12:29:51.957865 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:51.954313    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-99.ec2.internal\" not found"
Jul 17 12:29:52.083278 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.050001    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 12:29:52.083278 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:52.054607    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-99.ec2.internal\" not found"
Jul 17 12:29:52.093161 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.090633    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 12:29:52.211990 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.165095    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-99.ec2.internal"
Jul 17 12:29:52.211990 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.179814    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 12:29:52.211990 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.180824    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-99.ec2.internal"
Jul 17 12:29:52.211990 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.190674    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 12:29:52.424911 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.424885    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-99.ec2.internal"
Jul 17 12:29:52.429997 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.429979    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-99.ec2.internal"
Jul 17 12:29:52.440704 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.440685    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 12:29:52.440866 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:52.440787    2564 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-99.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-99.ec2.internal"
Jul 17 12:29:52.440946 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.440930    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 12:29:52.440988 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:52.440967    2564 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-99.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-99.ec2.internal"
Jul 17 12:29:52.444350 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.444334    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-99.ec2.internal"
Jul 17 12:29:52.445283 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.445269    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-99.ec2.internal"
Jul 17 12:29:52.655196 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.655103    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 12:29:52.748941 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.748903    2564 apiserver.go:52] "Watching apiserver"
Jul 17 12:29:52.755901 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.755882    2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 17 12:29:52.756289 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.756268    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-99.ec2.internal","openshift-network-diagnostics/network-check-target-xl2np","openshift-network-operator/iptables-alerter-xdqkr","openshift-ovn-kubernetes/ovnkube-node-9f958","kube-system/konnectivity-agent-5v7sc","kube-system/kube-apiserver-proxy-ip-10-0-137-99.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsvdx","openshift-dns/node-resolver-g8rzg","openshift-image-registry/node-ca-xxld5","openshift-multus/multus-7ql7p","openshift-multus/multus-additional-cni-plugins-sld25","openshift-multus/network-metrics-daemon-lxrxw","kube-system/global-pull-secret-syncer-9b4b4","openshift-cluster-node-tuning-operator/tuned-vtsx2"]
Jul 17 12:29:52.759125 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.759109    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.761459 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.761441    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsvdx"
Jul 17 12:29:52.762013 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.761994    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 17 12:29:52.762013 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.761990    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 17 12:29:52.762182 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.762040    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 17 12:29:52.762182 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.762053    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 17 12:29:52.762182 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.762062    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9w4rc\""
Jul 17 12:29:52.763801 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.763782    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xdqkr"
Jul 17 12:29:52.764065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.763989    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 17 12:29:52.764065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.764012    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z4wlg\""
Jul 17 12:29:52.764264 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.764169    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 17 12:29:52.764264 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.764202    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 17 12:29:52.765134 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.765116    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 17 12:29:52.766152 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.766134    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 17 12:29:52.766394 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.766362    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.766487 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.766424    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 17 12:29:52.766560 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.766424    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 17 12:29:52.766617 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.766513    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xgbr9\""
Jul 17 12:29:52.768755 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.768736    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jtdg9\""
Jul 17 12:29:52.768919 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.768904    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5v7sc"
Jul 17 12:29:52.769265 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.769249    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 17 12:29:52.769328 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.769251    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 17 12:29:52.771439 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.771419    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:29:52.771539 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:52.771508    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxrxw" podUID="937e7356-528f-4a56-a80e-9486657ec301"
Jul 17 12:29:52.771605 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.771525    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 17 12:29:52.771605 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.771584    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 17 12:29:52.771755 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.771740    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-b4b5j\""
Jul 17 12:29:52.774584 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.774276    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:29:52.774584 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:52.774394    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b4b4" podUID="c803ba62-1a8b-4b71-a355-fc4427be817b"
Jul 17 12:29:52.777031 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.776965    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-etc-modprobe-d\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.777366 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.777058    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-etc-sysconfig\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.777409 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.777395    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-etc-sysctl-conf\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.777443 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.777414    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-var-lib-kubelet\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.777488 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.777437    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-hostroot\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.777488 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.777463    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/83656ae5-efe4-426a-9a4c-080b18541deb-registration-dir\") pod \"aws-ebs-csi-driver-node-hsvdx\" (UID: \"83656ae5-efe4-426a-9a4c-080b18541deb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsvdx"
Jul 17 12:29:52.777578 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.777488    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/83656ae5-efe4-426a-9a4c-080b18541deb-sys-fs\") pod \"aws-ebs-csi-driver-node-hsvdx\" (UID: \"83656ae5-efe4-426a-9a4c-080b18541deb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsvdx"
Jul 17 12:29:52.777578 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.777511    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-host-run-multus-certs\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.777578 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.777534    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-etc-kubernetes\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.777578 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.777558    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/83656ae5-efe4-426a-9a4c-080b18541deb-device-dir\") pod \"aws-ebs-csi-driver-node-hsvdx\" (UID: \"83656ae5-efe4-426a-9a4c-080b18541deb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsvdx"
Jul 17 12:29:52.777749 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.777582    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/83656ae5-efe4-426a-9a4c-080b18541deb-etc-selinux\") pod \"aws-ebs-csi-driver-node-hsvdx\" (UID: \"83656ae5-efe4-426a-9a4c-080b18541deb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsvdx"
Jul 17 12:29:52.777749 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.777604    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe8942fd-34dd-4b5b-ba05-0c6488374820-host-slash\") pod \"iptables-alerter-xdqkr\" (UID: \"fe8942fd-34dd-4b5b-ba05-0c6488374820\") " pod="openshift-network-operator/iptables-alerter-xdqkr"
Jul 17 12:29:52.777749 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.777629    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-etc-sysctl-d\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.777749 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.777674    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-run\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.777749 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.777723    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-sys\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.778000 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.777753    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvjrb\" (UniqueName: \"kubernetes.io/projected/f92fb403-4621-413f-9e6b-c041411f774a-kube-api-access-lvjrb\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.778000 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.777785    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-system-cni-dir\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.778000 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.777843    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-cnibin\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.778000 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.777878    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-etc-systemd\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.778000 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.777904    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c47b98e9-a381-4815-b8d9-4737cf123e0d-cni-binary-copy\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.778000 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.777930    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-host-run-k8s-cni-cncf-io\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.778000 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.777969    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-multus-conf-dir\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.778000 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.777996    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/83656ae5-efe4-426a-9a4c-080b18541deb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hsvdx\" (UID: \"83656ae5-efe4-426a-9a4c-080b18541deb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsvdx"
Jul 17 12:29:52.778370 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.778021    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/83656ae5-efe4-426a-9a4c-080b18541deb-socket-dir\") pod \"aws-ebs-csi-driver-node-hsvdx\" (UID: \"83656ae5-efe4-426a-9a4c-080b18541deb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsvdx"
Jul 17 12:29:52.778370 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.778055    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fe8942fd-34dd-4b5b-ba05-0c6488374820-iptables-alerter-script\") pod \"iptables-alerter-xdqkr\" (UID: \"fe8942fd-34dd-4b5b-ba05-0c6488374820\") " pod="openshift-network-operator/iptables-alerter-xdqkr"
Jul 17 12:29:52.778370 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.778095    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhz54\" (UniqueName: \"kubernetes.io/projected/fe8942fd-34dd-4b5b-ba05-0c6488374820-kube-api-access-dhz54\") pod \"iptables-alerter-xdqkr\" (UID: \"fe8942fd-34dd-4b5b-ba05-0c6488374820\") " pod="openshift-network-operator/iptables-alerter-xdqkr"
Jul 17 12:29:52.778370 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.778120    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-etc-kubernetes\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.778370 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.778160    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-host\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.778370 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.778180    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xxld5"
Jul 17 12:29:52.778370 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.778204    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f92fb403-4621-413f-9e6b-c041411f774a-etc-tuned\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.778370 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.778235    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f92fb403-4621-413f-9e6b-c041411f774a-tmp\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.778370 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.778277    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-multus-socket-dir-parent\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.778370 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.778305    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-host-run-netns\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.778370 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.778334    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-host-var-lib-cni-bin\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.778370 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.778373    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-host-var-lib-kubelet\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.778846 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.778399    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c47b98e9-a381-4815-b8d9-4737cf123e0d-multus-daemon-config\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.778846 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.778436    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvfxl\" (UniqueName: \"kubernetes.io/projected/83656ae5-efe4-426a-9a4c-080b18541deb-kube-api-access-wvfxl\") pod \"aws-ebs-csi-driver-node-hsvdx\" (UID: \"83656ae5-efe4-426a-9a4c-080b18541deb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsvdx"
Jul 17 12:29:52.778846 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.778479    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-lib-modules\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.778846 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.778497    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-multus-cni-dir\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.778846 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.778511    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-os-release\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.778846 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.778542    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-host-var-lib-cni-multus\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.778846 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.778571    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6rl4\" (UniqueName: \"kubernetes.io/projected/c47b98e9-a381-4815-b8d9-4737cf123e0d-kube-api-access-z6rl4\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.780697 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.780677    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:29:52.780807 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:52.780784    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xl2np" podUID="a72fcab3-1981-4637-a041-225335390f1d"
Jul 17 12:29:52.781210 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.781192    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 17 12:29:52.781292 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.781248    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 17 12:29:52.781292 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.781273    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w668q\""
Jul 17 12:29:52.781564 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.781540    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 12:29:52.781635 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.781570    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 17 12:29:52.783268 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.783251    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sld25"
Jul 17 12:29:52.785635 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.785617    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g8rzg"
Jul 17 12:29:52.785788 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.785759    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 17 12:29:52.785865 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.785852    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-948cp\""
Jul 17 12:29:52.785965 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.785949    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 17 12:29:52.788128 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.788110    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.788458 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.788439    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 17 12:29:52.788788 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.788772    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 17 12:29:52.788855 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.788805    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v9zzn\""
Jul 17 12:29:52.791084 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.791063    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 17 12:29:52.791194 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.791178    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 17 12:29:52.791395 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.791383    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 17 12:29:52.791437 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.791384    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 17 12:29:52.791583 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.791569    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 17 12:29:52.791583 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.791579    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 17 12:29:52.791672 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.791639    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-g9jxh\""
Jul 17 12:29:52.804126 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.804105    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xplpq"
Jul 17 12:29:52.812923 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.812902    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xplpq"
Jul 17 12:29:52.865963 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.865941    2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 17 12:29:52.879580 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.879557    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/83656ae5-efe4-426a-9a4c-080b18541deb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hsvdx\" (UID: \"83656ae5-efe4-426a-9a4c-080b18541deb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsvdx"
Jul 17 12:29:52.879738 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.879593    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-host-kubelet\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.879738 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.879616    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-etc-openvswitch\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.879738 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.879640    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z8tx\" (UniqueName: \"kubernetes.io/projected/cc19f981-8855-47ce-98c7-6e8e9c0e65d2-kube-api-access-4z8tx\") pod \"node-ca-xxld5\" (UID: \"cc19f981-8855-47ce-98c7-6e8e9c0e65d2\") " pod="openshift-image-registry/node-ca-xxld5"
Jul 17 12:29:52.879738 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.879665    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-etc-kubernetes\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.879738 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.879687    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f92fb403-4621-413f-9e6b-c041411f774a-etc-tuned\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.879738 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.879711    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f92fb403-4621-413f-9e6b-c041411f774a-tmp\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.880171 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.879757    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntrg7\" (UniqueName: \"kubernetes.io/projected/937e7356-528f-4a56-a80e-9486657ec301-kube-api-access-ntrg7\") pod \"network-metrics-daemon-lxrxw\" (UID: \"937e7356-528f-4a56-a80e-9486657ec301\") " pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:29:52.880171 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.879791    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/83656ae5-efe4-426a-9a4c-080b18541deb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hsvdx\" (UID: \"83656ae5-efe4-426a-9a4c-080b18541deb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsvdx"
Jul 17 12:29:52.880171 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.879803    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-etc-kubernetes\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.880171 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.879806    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7ab622df-86d3-4cc3-aba7-123f4bc0c996-env-overrides\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.880171 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.879860    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-host-var-lib-cni-bin\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.880171 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.879891    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-run-systemd\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.880171 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.879950    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-host-var-lib-cni-bin\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.880171 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.879969    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc19f981-8855-47ce-98c7-6e8e9c0e65d2-serviceca\") pod \"node-ca-xxld5\" (UID: \"cc19f981-8855-47ce-98c7-6e8e9c0e65d2\") " pod="openshift-image-registry/node-ca-xxld5"
Jul 17 12:29:52.880171 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.880023    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjgdr\" (UniqueName: \"kubernetes.io/projected/b09e01d5-4dca-4a57-903d-f6daacfdb5d9-kube-api-access-wjgdr\") pod \"multus-additional-cni-plugins-sld25\" (UID: \"b09e01d5-4dca-4a57-903d-f6daacfdb5d9\") " pod="openshift-multus/multus-additional-cni-plugins-sld25"
Jul 17 12:29:52.880171 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.880067    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-os-release\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.880171 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.880122    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6rl4\" (UniqueName: \"kubernetes.io/projected/c47b98e9-a381-4815-b8d9-4737cf123e0d-kube-api-access-z6rl4\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.880719 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.880181    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-os-release\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.880719 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.880176    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ab622df-86d3-4cc3-aba7-123f4bc0c996-ovn-node-metrics-cert\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.880719 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.880258    2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 17 12:29:52.880719 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.880278    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/688df482-0ac3-4a72-9632-c6a8c18b0b8d-tmp-dir\") pod \"node-resolver-g8rzg\" (UID: \"688df482-0ac3-4a72-9632-c6a8c18b0b8d\") " pod="openshift-dns/node-resolver-g8rzg"
Jul 17 12:29:52.880719 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.880325    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc19f981-8855-47ce-98c7-6e8e9c0e65d2-host\") pod \"node-ca-xxld5\" (UID: \"cc19f981-8855-47ce-98c7-6e8e9c0e65d2\") " pod="openshift-image-registry/node-ca-xxld5"
Jul 17 12:29:52.880719 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.880361    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmwjl\" (UniqueName: \"kubernetes.io/projected/a72fcab3-1981-4637-a041-225335390f1d-kube-api-access-pmwjl\") pod \"network-check-target-xl2np\" (UID: \"a72fcab3-1981-4637-a041-225335390f1d\") " pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:29:52.880719 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.880398    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-etc-sysconfig\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.880719 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.880431    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-etc-sysctl-conf\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.880719 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.880469    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-hostroot\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.880719 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.880467    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-etc-sysconfig\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.880719 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.880501    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/83656ae5-efe4-426a-9a4c-080b18541deb-registration-dir\") pod \"aws-ebs-csi-driver-node-hsvdx\" (UID: \"83656ae5-efe4-426a-9a4c-080b18541deb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsvdx"
Jul 17 12:29:52.880719 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.880529    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-hostroot\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.880719 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.880563    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-node-log\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.880719 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.880593    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.880719 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.880604    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/83656ae5-efe4-426a-9a4c-080b18541deb-registration-dir\") pod \"aws-ebs-csi-driver-node-hsvdx\" (UID: \"83656ae5-efe4-426a-9a4c-080b18541deb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsvdx"
Jul 17 12:29:52.880719 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.880657    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/937e7356-528f-4a56-a80e-9486657ec301-metrics-certs\") pod \"network-metrics-daemon-lxrxw\" (UID: \"937e7356-528f-4a56-a80e-9486657ec301\") " pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:29:52.881689 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.880790    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-etc-sysctl-conf\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.881689 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.880779    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b09e01d5-4dca-4a57-903d-f6daacfdb5d9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sld25\" (UID: \"b09e01d5-4dca-4a57-903d-f6daacfdb5d9\") " pod="openshift-multus/multus-additional-cni-plugins-sld25"
Jul 17 12:29:52.881689 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.880902    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-host-run-multus-certs\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.881689 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.880944    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-etc-kubernetes\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.881689 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.880972    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/83656ae5-efe4-426a-9a4c-080b18541deb-device-dir\") pod \"aws-ebs-csi-driver-node-hsvdx\" (UID: \"83656ae5-efe4-426a-9a4c-080b18541deb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsvdx"
Jul 17 12:29:52.881689 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.881000    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b09e01d5-4dca-4a57-903d-f6daacfdb5d9-cni-binary-copy\") pod \"multus-additional-cni-plugins-sld25\" (UID: \"b09e01d5-4dca-4a57-903d-f6daacfdb5d9\") " pod="openshift-multus/multus-additional-cni-plugins-sld25"
Jul 17 12:29:52.881689 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.881026    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b09e01d5-4dca-4a57-903d-f6daacfdb5d9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sld25\" (UID: \"b09e01d5-4dca-4a57-903d-f6daacfdb5d9\") " pod="openshift-multus/multus-additional-cni-plugins-sld25"
Jul 17 12:29:52.881689 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.881027    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-etc-kubernetes\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.882189 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.881056    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-etc-sysctl-d\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.882189 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.882080    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-host-run-multus-certs\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.883298 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.882784    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/83656ae5-efe4-426a-9a4c-080b18541deb-device-dir\") pod \"aws-ebs-csi-driver-node-hsvdx\" (UID: \"83656ae5-efe4-426a-9a4c-080b18541deb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsvdx"
Jul 17 12:29:52.883298 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.883118    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-sys\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.883298 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.883205    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-etc-sysctl-d\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.885833 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.885690    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-system-cni-dir\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.885833 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.885735    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-cnibin\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.885833 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.885794    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-systemd-units\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.885833 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.885814    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-system-cni-dir\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.885833 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.885824    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f92fb403-4621-413f-9e6b-c041411f774a-tmp\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.886133 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.885844    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f92fb403-4621-413f-9e6b-c041411f774a-etc-tuned\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.886133 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.885884    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-host-run-ovn-kubernetes\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.886133 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.885949    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-sys\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.886133 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.885965    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-cnibin\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.886133 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.885987    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/146d6cf8-52ca-4482-9abb-ef40e770eb13-agent-certs\") pod \"konnectivity-agent-5v7sc\" (UID: \"146d6cf8-52ca-4482-9abb-ef40e770eb13\") " pod="kube-system/konnectivity-agent-5v7sc"
Jul 17 12:29:52.886133 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.886060    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-multus-conf-dir\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.886133 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.886076    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-multus-conf-dir\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.886133 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.886116    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhz54\" (UniqueName: \"kubernetes.io/projected/fe8942fd-34dd-4b5b-ba05-0c6488374820-kube-api-access-dhz54\") pod \"iptables-alerter-xdqkr\" (UID: \"fe8942fd-34dd-4b5b-ba05-0c6488374820\") " pod="openshift-network-operator/iptables-alerter-xdqkr"
Jul 17 12:29:52.886392 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.886169    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-var-lib-openvswitch\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.886623 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.886595    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-host\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.886727 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.886610    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-host\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.886727 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.886639    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-multus-socket-dir-parent\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.886727 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.886658    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-host-run-netns\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.886727 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.886677    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-host-run-netns\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.886727 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.886693    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7ab622df-86d3-4cc3-aba7-123f4bc0c996-ovnkube-script-lib\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.886727 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.886710    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c803ba62-1a8b-4b71-a355-fc4427be817b-kubelet-config\") pod \"global-pull-secret-syncer-9b4b4\" (UID: \"c803ba62-1a8b-4b71-a355-fc4427be817b\") " pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:29:52.886727 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.886727    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b09e01d5-4dca-4a57-903d-f6daacfdb5d9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sld25\" (UID: \"b09e01d5-4dca-4a57-903d-f6daacfdb5d9\") " pod="openshift-multus/multus-additional-cni-plugins-sld25"
Jul 17 12:29:52.887164 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.886752    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-host-run-netns\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.887164 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.886747    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-run-openvswitch\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.887164 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.886800    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7gd8\" (UniqueName: \"kubernetes.io/projected/688df482-0ac3-4a72-9632-c6a8c18b0b8d-kube-api-access-r7gd8\") pod \"node-resolver-g8rzg\" (UID: \"688df482-0ac3-4a72-9632-c6a8c18b0b8d\") " pod="openshift-dns/node-resolver-g8rzg"
Jul 17 12:29:52.887164 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.886837    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-host-var-lib-kubelet\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.887164 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.886847    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-multus-socket-dir-parent\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.887164 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.886868    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c47b98e9-a381-4815-b8d9-4737cf123e0d-multus-daemon-config\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.887164 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.886891    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-host-var-lib-kubelet\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.887164 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.886892    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvfxl\" (UniqueName: \"kubernetes.io/projected/83656ae5-efe4-426a-9a4c-080b18541deb-kube-api-access-wvfxl\") pod \"aws-ebs-csi-driver-node-hsvdx\" (UID: \"83656ae5-efe4-426a-9a4c-080b18541deb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsvdx"
Jul 17 12:29:52.887164 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.886917    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/688df482-0ac3-4a72-9632-c6a8c18b0b8d-hosts-file\") pod \"node-resolver-g8rzg\" (UID: \"688df482-0ac3-4a72-9632-c6a8c18b0b8d\") " pod="openshift-dns/node-resolver-g8rzg"
Jul 17 12:29:52.887164 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.886934    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b09e01d5-4dca-4a57-903d-f6daacfdb5d9-system-cni-dir\") pod \"multus-additional-cni-plugins-sld25\" (UID: \"b09e01d5-4dca-4a57-903d-f6daacfdb5d9\") " pod="openshift-multus/multus-additional-cni-plugins-sld25"
Jul 17 12:29:52.887164 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.886960    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-lib-modules\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.887164 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.886984    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-multus-cni-dir\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.887164 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887003    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-host-var-lib-cni-multus\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.887164 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887021    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-run-ovn\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.887164 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887042    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-log-socket\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.887164 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887056    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-lib-modules\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.887164 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887106    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-host-var-lib-cni-multus\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.888127 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887146    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-host-cni-bin\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.888127 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887150    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-multus-cni-dir\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.888127 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887182    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/146d6cf8-52ca-4482-9abb-ef40e770eb13-konnectivity-ca\") pod \"konnectivity-agent-5v7sc\" (UID: \"146d6cf8-52ca-4482-9abb-ef40e770eb13\") " pod="kube-system/konnectivity-agent-5v7sc"
Jul 17 12:29:52.888127 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887218    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-etc-modprobe-d\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.888127 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887242    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-var-lib-kubelet\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.888127 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887266    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/83656ae5-efe4-426a-9a4c-080b18541deb-sys-fs\") pod \"aws-ebs-csi-driver-node-hsvdx\" (UID: \"83656ae5-efe4-426a-9a4c-080b18541deb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsvdx"
Jul 17 12:29:52.888127 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887290    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-host-cni-netd\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.888127 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887312    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-var-lib-kubelet\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.888127 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887312    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn2b9\" (UniqueName: \"kubernetes.io/projected/7ab622df-86d3-4cc3-aba7-123f4bc0c996-kube-api-access-dn2b9\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.888127 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887339    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-etc-modprobe-d\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.888127 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887357    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/83656ae5-efe4-426a-9a4c-080b18541deb-sys-fs\") pod \"aws-ebs-csi-driver-node-hsvdx\" (UID: \"83656ae5-efe4-426a-9a4c-080b18541deb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsvdx"
Jul 17 12:29:52.888127 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887355    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c803ba62-1a8b-4b71-a355-fc4427be817b-dbus\") pod \"global-pull-secret-syncer-9b4b4\" (UID: \"c803ba62-1a8b-4b71-a355-fc4427be817b\") " pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:29:52.888127 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887385    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c803ba62-1a8b-4b71-a355-fc4427be817b-original-pull-secret\") pod \"global-pull-secret-syncer-9b4b4\" (UID: \"c803ba62-1a8b-4b71-a355-fc4427be817b\") " pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:29:52.888127 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887410    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/83656ae5-efe4-426a-9a4c-080b18541deb-etc-selinux\") pod \"aws-ebs-csi-driver-node-hsvdx\" (UID: \"83656ae5-efe4-426a-9a4c-080b18541deb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsvdx"
Jul 17 12:29:52.888127 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887428    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe8942fd-34dd-4b5b-ba05-0c6488374820-host-slash\") pod \"iptables-alerter-xdqkr\" (UID: \"fe8942fd-34dd-4b5b-ba05-0c6488374820\") " pod="openshift-network-operator/iptables-alerter-xdqkr"
Jul 17 12:29:52.888127 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887443    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-host-slash\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.888127 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887456    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7ab622df-86d3-4cc3-aba7-123f4bc0c996-ovnkube-config\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.889021 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887470    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b09e01d5-4dca-4a57-903d-f6daacfdb5d9-cnibin\") pod \"multus-additional-cni-plugins-sld25\" (UID: \"b09e01d5-4dca-4a57-903d-f6daacfdb5d9\") " pod="openshift-multus/multus-additional-cni-plugins-sld25"
Jul 17 12:29:52.889021 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887470    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/83656ae5-efe4-426a-9a4c-080b18541deb-etc-selinux\") pod \"aws-ebs-csi-driver-node-hsvdx\" (UID: \"83656ae5-efe4-426a-9a4c-080b18541deb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsvdx"
Jul 17 12:29:52.889021 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887494    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-run\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.889021 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887514    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvjrb\" (UniqueName: \"kubernetes.io/projected/f92fb403-4621-413f-9e6b-c041411f774a-kube-api-access-lvjrb\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.889021 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887516    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c47b98e9-a381-4815-b8d9-4737cf123e0d-multus-daemon-config\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.889021 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887528    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c47b98e9-a381-4815-b8d9-4737cf123e0d-cni-binary-copy\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.889021 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887543    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/83656ae5-efe4-426a-9a4c-080b18541deb-socket-dir\") pod \"aws-ebs-csi-driver-node-hsvdx\" (UID: \"83656ae5-efe4-426a-9a4c-080b18541deb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsvdx"
Jul 17 12:29:52.889021 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887562    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe8942fd-34dd-4b5b-ba05-0c6488374820-host-slash\") pod \"iptables-alerter-xdqkr\" (UID: \"fe8942fd-34dd-4b5b-ba05-0c6488374820\") " pod="openshift-network-operator/iptables-alerter-xdqkr"
Jul 17 12:29:52.889021 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887610    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-run\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.889021 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887727    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/83656ae5-efe4-426a-9a4c-080b18541deb-socket-dir\") pod \"aws-ebs-csi-driver-node-hsvdx\" (UID: \"83656ae5-efe4-426a-9a4c-080b18541deb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsvdx"
Jul 17 12:29:52.889021 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887781    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fe8942fd-34dd-4b5b-ba05-0c6488374820-iptables-alerter-script\") pod \"iptables-alerter-xdqkr\" (UID: \"fe8942fd-34dd-4b5b-ba05-0c6488374820\") " pod="openshift-network-operator/iptables-alerter-xdqkr"
Jul 17 12:29:52.889021 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887808    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b09e01d5-4dca-4a57-903d-f6daacfdb5d9-os-release\") pod \"multus-additional-cni-plugins-sld25\" (UID: \"b09e01d5-4dca-4a57-903d-f6daacfdb5d9\") " pod="openshift-multus/multus-additional-cni-plugins-sld25"
Jul 17 12:29:52.889021 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887827    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-etc-systemd\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.889021 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887843    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-host-run-k8s-cni-cncf-io\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.889021 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887928    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c47b98e9-a381-4815-b8d9-4737cf123e0d-host-run-k8s-cni-cncf-io\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.889021 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.887971    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f92fb403-4621-413f-9e6b-c041411f774a-etc-systemd\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.889021 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.888156    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c47b98e9-a381-4815-b8d9-4737cf123e0d-cni-binary-copy\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.889966 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.888316    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fe8942fd-34dd-4b5b-ba05-0c6488374820-iptables-alerter-script\") pod \"iptables-alerter-xdqkr\" (UID: \"fe8942fd-34dd-4b5b-ba05-0c6488374820\") " pod="openshift-network-operator/iptables-alerter-xdqkr"
Jul 17 12:29:52.893326 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.893298    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6rl4\" (UniqueName: \"kubernetes.io/projected/c47b98e9-a381-4815-b8d9-4737cf123e0d-kube-api-access-z6rl4\") pod \"multus-7ql7p\" (UID: \"c47b98e9-a381-4815-b8d9-4737cf123e0d\") " pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:52.895536 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.895509    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhz54\" (UniqueName: \"kubernetes.io/projected/fe8942fd-34dd-4b5b-ba05-0c6488374820-kube-api-access-dhz54\") pod \"iptables-alerter-xdqkr\" (UID: \"fe8942fd-34dd-4b5b-ba05-0c6488374820\") " pod="openshift-network-operator/iptables-alerter-xdqkr"
Jul 17 12:29:52.895731 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.895702    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvjrb\" (UniqueName: \"kubernetes.io/projected/f92fb403-4621-413f-9e6b-c041411f774a-kube-api-access-lvjrb\") pod \"tuned-vtsx2\" (UID: \"f92fb403-4621-413f-9e6b-c041411f774a\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:52.895955 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.895937    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvfxl\" (UniqueName: \"kubernetes.io/projected/83656ae5-efe4-426a-9a4c-080b18541deb-kube-api-access-wvfxl\") pod \"aws-ebs-csi-driver-node-hsvdx\" (UID: \"83656ae5-efe4-426a-9a4c-080b18541deb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsvdx"
Jul 17 12:29:52.989092 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989053    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-host-run-ovn-kubernetes\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.989414 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989100    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/146d6cf8-52ca-4482-9abb-ef40e770eb13-agent-certs\") pod \"konnectivity-agent-5v7sc\" (UID: \"146d6cf8-52ca-4482-9abb-ef40e770eb13\") " pod="kube-system/konnectivity-agent-5v7sc"
Jul 17 12:29:52.989414 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989151    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-var-lib-openvswitch\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.989414 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989196    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-host-run-netns\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.989414 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989226    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7ab622df-86d3-4cc3-aba7-123f4bc0c996-ovnkube-script-lib\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.989414 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989249    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-var-lib-openvswitch\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.989414 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989258    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c803ba62-1a8b-4b71-a355-fc4427be817b-kubelet-config\") pod \"global-pull-secret-syncer-9b4b4\" (UID: \"c803ba62-1a8b-4b71-a355-fc4427be817b\") " pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:29:52.989414 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989301    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-host-run-netns\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.989414 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989321    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c803ba62-1a8b-4b71-a355-fc4427be817b-kubelet-config\") pod \"global-pull-secret-syncer-9b4b4\" (UID: \"c803ba62-1a8b-4b71-a355-fc4427be817b\") " pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:29:52.989414 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989337    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b09e01d5-4dca-4a57-903d-f6daacfdb5d9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sld25\" (UID: \"b09e01d5-4dca-4a57-903d-f6daacfdb5d9\") " pod="openshift-multus/multus-additional-cni-plugins-sld25"
Jul 17 12:29:52.989414 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989372    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-host-run-ovn-kubernetes\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.989414 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989387    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-run-openvswitch\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.989414 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989413    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7gd8\" (UniqueName: \"kubernetes.io/projected/688df482-0ac3-4a72-9632-c6a8c18b0b8d-kube-api-access-r7gd8\") pod \"node-resolver-g8rzg\" (UID: \"688df482-0ac3-4a72-9632-c6a8c18b0b8d\") " pod="openshift-dns/node-resolver-g8rzg"
Jul 17 12:29:52.990140 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989454    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-run-openvswitch\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.990140 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989444    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/688df482-0ac3-4a72-9632-c6a8c18b0b8d-hosts-file\") pod \"node-resolver-g8rzg\" (UID: \"688df482-0ac3-4a72-9632-c6a8c18b0b8d\") " pod="openshift-dns/node-resolver-g8rzg"
Jul 17 12:29:52.990140 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989497    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b09e01d5-4dca-4a57-903d-f6daacfdb5d9-system-cni-dir\") pod \"multus-additional-cni-plugins-sld25\" (UID: \"b09e01d5-4dca-4a57-903d-f6daacfdb5d9\") " pod="openshift-multus/multus-additional-cni-plugins-sld25"
Jul 17 12:29:52.990140 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989529    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-run-ovn\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.990140 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989555    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-log-socket\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.990140 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989582    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-host-cni-bin\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.990140 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989610    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/688df482-0ac3-4a72-9632-c6a8c18b0b8d-hosts-file\") pod \"node-resolver-g8rzg\" (UID: \"688df482-0ac3-4a72-9632-c6a8c18b0b8d\") " pod="openshift-dns/node-resolver-g8rzg"
Jul 17 12:29:52.990140 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989640    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/146d6cf8-52ca-4482-9abb-ef40e770eb13-konnectivity-ca\") pod \"konnectivity-agent-5v7sc\" (UID: \"146d6cf8-52ca-4482-9abb-ef40e770eb13\") " pod="kube-system/konnectivity-agent-5v7sc"
Jul 17 12:29:52.990140 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989685    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-host-cni-netd\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.990140 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989710    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn2b9\" (UniqueName: \"kubernetes.io/projected/7ab622df-86d3-4cc3-aba7-123f4bc0c996-kube-api-access-dn2b9\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.990140 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989727    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b09e01d5-4dca-4a57-903d-f6daacfdb5d9-system-cni-dir\") pod \"multus-additional-cni-plugins-sld25\" (UID: \"b09e01d5-4dca-4a57-903d-f6daacfdb5d9\") " pod="openshift-multus/multus-additional-cni-plugins-sld25"
Jul 17 12:29:52.990140 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989787    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c803ba62-1a8b-4b71-a355-fc4427be817b-dbus\") pod \"global-pull-secret-syncer-9b4b4\" (UID: \"c803ba62-1a8b-4b71-a355-fc4427be817b\") " pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:29:52.990140 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989819    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c803ba62-1a8b-4b71-a355-fc4427be817b-original-pull-secret\") pod \"global-pull-secret-syncer-9b4b4\" (UID: \"c803ba62-1a8b-4b71-a355-fc4427be817b\") " pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:29:52.990140 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989849    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-log-socket\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.990140 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989867    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-host-slash\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.990140 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989932    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7ab622df-86d3-4cc3-aba7-123f4bc0c996-ovnkube-config\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.990140 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989936    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-host-cni-bin\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.991015 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989974    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b09e01d5-4dca-4a57-903d-f6daacfdb5d9-cnibin\") pod \"multus-additional-cni-plugins-sld25\" (UID: \"b09e01d5-4dca-4a57-903d-f6daacfdb5d9\") " pod="openshift-multus/multus-additional-cni-plugins-sld25"
Jul 17 12:29:52.991015 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990016    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b09e01d5-4dca-4a57-903d-f6daacfdb5d9-os-release\") pod \"multus-additional-cni-plugins-sld25\" (UID: \"b09e01d5-4dca-4a57-903d-f6daacfdb5d9\") " pod="openshift-multus/multus-additional-cni-plugins-sld25"
Jul 17 12:29:52.991015 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990035    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7ab622df-86d3-4cc3-aba7-123f4bc0c996-ovnkube-script-lib\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.991015 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990051    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-host-kubelet\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.991015 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990089    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-etc-openvswitch\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.991015 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990100    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b09e01d5-4dca-4a57-903d-f6daacfdb5d9-cnibin\") pod \"multus-additional-cni-plugins-sld25\" (UID: \"b09e01d5-4dca-4a57-903d-f6daacfdb5d9\") " pod="openshift-multus/multus-additional-cni-plugins-sld25"
Jul 17 12:29:52.991015 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990132    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z8tx\" (UniqueName: \"kubernetes.io/projected/cc19f981-8855-47ce-98c7-6e8e9c0e65d2-kube-api-access-4z8tx\") pod \"node-ca-xxld5\" (UID: \"cc19f981-8855-47ce-98c7-6e8e9c0e65d2\") " pod="openshift-image-registry/node-ca-xxld5"
Jul 17 12:29:52.991015 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990141    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b09e01d5-4dca-4a57-903d-f6daacfdb5d9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sld25\" (UID: \"b09e01d5-4dca-4a57-903d-f6daacfdb5d9\") " pod="openshift-multus/multus-additional-cni-plugins-sld25"
Jul 17 12:29:52.991015 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990166    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-host-cni-netd\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.991015 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990177    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b09e01d5-4dca-4a57-903d-f6daacfdb5d9-os-release\") pod \"multus-additional-cni-plugins-sld25\" (UID: \"b09e01d5-4dca-4a57-903d-f6daacfdb5d9\") " pod="openshift-multus/multus-additional-cni-plugins-sld25"
Jul 17 12:29:52.991015 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990172    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntrg7\" (UniqueName: \"kubernetes.io/projected/937e7356-528f-4a56-a80e-9486657ec301-kube-api-access-ntrg7\") pod \"network-metrics-daemon-lxrxw\" (UID: \"937e7356-528f-4a56-a80e-9486657ec301\") " pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:29:52.991015 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.989685    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-run-ovn\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.991015 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990242    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-etc-openvswitch\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.991015 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990247    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7ab622df-86d3-4cc3-aba7-123f4bc0c996-env-overrides\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.991015 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990220    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-host-kubelet\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.991015 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990273    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c803ba62-1a8b-4b71-a355-fc4427be817b-dbus\") pod \"global-pull-secret-syncer-9b4b4\" (UID: \"c803ba62-1a8b-4b71-a355-fc4427be817b\") " pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:29:52.991015 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990325    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-run-systemd\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.991915 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990352    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc19f981-8855-47ce-98c7-6e8e9c0e65d2-serviceca\") pod \"node-ca-xxld5\" (UID: \"cc19f981-8855-47ce-98c7-6e8e9c0e65d2\") " pod="openshift-image-registry/node-ca-xxld5"
Jul 17 12:29:52.991915 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:52.990424    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 12:29:52.991915 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990438    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/146d6cf8-52ca-4482-9abb-ef40e770eb13-konnectivity-ca\") pod \"konnectivity-agent-5v7sc\" (UID: \"146d6cf8-52ca-4482-9abb-ef40e770eb13\") " pod="kube-system/konnectivity-agent-5v7sc"
Jul 17 12:29:52.991915 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990455    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7ab622df-86d3-4cc3-aba7-123f4bc0c996-ovnkube-config\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.991915 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990476    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjgdr\" (UniqueName: \"kubernetes.io/projected/b09e01d5-4dca-4a57-903d-f6daacfdb5d9-kube-api-access-wjgdr\") pod \"multus-additional-cni-plugins-sld25\" (UID: \"b09e01d5-4dca-4a57-903d-f6daacfdb5d9\") " pod="openshift-multus/multus-additional-cni-plugins-sld25"
Jul 17 12:29:52.991915 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990513    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-host-slash\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.991915 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990483    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-run-systemd\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.991915 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:52.990564    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c803ba62-1a8b-4b71-a355-fc4427be817b-original-pull-secret podName:c803ba62-1a8b-4b71-a355-fc4427be817b nodeName:}" failed. No retries permitted until 2026-07-17 12:29:53.490525303 +0000 UTC m=+3.070637322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c803ba62-1a8b-4b71-a355-fc4427be817b-original-pull-secret") pod "global-pull-secret-syncer-9b4b4" (UID: "c803ba62-1a8b-4b71-a355-fc4427be817b") : object "kube-system"/"original-pull-secret" not registered
Jul 17 12:29:52.991915 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990677    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ab622df-86d3-4cc3-aba7-123f4bc0c996-ovn-node-metrics-cert\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.991915 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990720    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/688df482-0ac3-4a72-9632-c6a8c18b0b8d-tmp-dir\") pod \"node-resolver-g8rzg\" (UID: \"688df482-0ac3-4a72-9632-c6a8c18b0b8d\") " pod="openshift-dns/node-resolver-g8rzg"
Jul 17 12:29:52.991915 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990744    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc19f981-8855-47ce-98c7-6e8e9c0e65d2-host\") pod \"node-ca-xxld5\" (UID: \"cc19f981-8855-47ce-98c7-6e8e9c0e65d2\") " pod="openshift-image-registry/node-ca-xxld5"
Jul 17 12:29:52.991915 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990795    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmwjl\" (UniqueName: \"kubernetes.io/projected/a72fcab3-1981-4637-a041-225335390f1d-kube-api-access-pmwjl\") pod \"network-check-target-xl2np\" (UID: \"a72fcab3-1981-4637-a041-225335390f1d\") " pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:29:52.991915 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990839    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-node-log\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.991915 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990875    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.991915 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990903    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/937e7356-528f-4a56-a80e-9486657ec301-metrics-certs\") pod \"network-metrics-daemon-lxrxw\" (UID: \"937e7356-528f-4a56-a80e-9486657ec301\") " pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:29:52.991915 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990906    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7ab622df-86d3-4cc3-aba7-123f4bc0c996-env-overrides\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.991915 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990934    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc19f981-8855-47ce-98c7-6e8e9c0e65d2-serviceca\") pod \"node-ca-xxld5\" (UID: \"cc19f981-8855-47ce-98c7-6e8e9c0e65d2\") " pod="openshift-image-registry/node-ca-xxld5"
Jul 17 12:29:52.992476 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990934    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b09e01d5-4dca-4a57-903d-f6daacfdb5d9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sld25\" (UID: \"b09e01d5-4dca-4a57-903d-f6daacfdb5d9\") " pod="openshift-multus/multus-additional-cni-plugins-sld25"
Jul 17 12:29:52.992476 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.990990    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.992476 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.991041    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-node-log\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.992476 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.991108    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc19f981-8855-47ce-98c7-6e8e9c0e65d2-host\") pod \"node-ca-xxld5\" (UID: \"cc19f981-8855-47ce-98c7-6e8e9c0e65d2\") " pod="openshift-image-registry/node-ca-xxld5"
Jul 17 12:29:52.992476 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:52.991134    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:29:52.992476 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.991195    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b09e01d5-4dca-4a57-903d-f6daacfdb5d9-cni-binary-copy\") pod \"multus-additional-cni-plugins-sld25\" (UID: \"b09e01d5-4dca-4a57-903d-f6daacfdb5d9\") " pod="openshift-multus/multus-additional-cni-plugins-sld25"
Jul 17 12:29:52.992476 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.991223    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b09e01d5-4dca-4a57-903d-f6daacfdb5d9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sld25\" (UID: \"b09e01d5-4dca-4a57-903d-f6daacfdb5d9\") " pod="openshift-multus/multus-additional-cni-plugins-sld25"
Jul 17 12:29:52.992476 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.991255    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-systemd-units\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.992476 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.991326    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7ab622df-86d3-4cc3-aba7-123f4bc0c996-systemd-units\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:52.992476 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:52.991362    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/937e7356-528f-4a56-a80e-9486657ec301-metrics-certs podName:937e7356-528f-4a56-a80e-9486657ec301 nodeName:}" failed. No retries permitted until 2026-07-17 12:29:53.491344681 +0000 UTC m=+3.071456695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/937e7356-528f-4a56-a80e-9486657ec301-metrics-certs") pod "network-metrics-daemon-lxrxw" (UID: "937e7356-528f-4a56-a80e-9486657ec301") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:29:52.992476 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.991400    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b09e01d5-4dca-4a57-903d-f6daacfdb5d9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sld25\" (UID: \"b09e01d5-4dca-4a57-903d-f6daacfdb5d9\") " pod="openshift-multus/multus-additional-cni-plugins-sld25"
Jul 17 12:29:52.992476 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.991622    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b09e01d5-4dca-4a57-903d-f6daacfdb5d9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sld25\" (UID: \"b09e01d5-4dca-4a57-903d-f6daacfdb5d9\") " pod="openshift-multus/multus-additional-cni-plugins-sld25"
Jul 17 12:29:52.992476 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.991794    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b09e01d5-4dca-4a57-903d-f6daacfdb5d9-cni-binary-copy\") pod \"multus-additional-cni-plugins-sld25\" (UID: \"b09e01d5-4dca-4a57-903d-f6daacfdb5d9\") " pod="openshift-multus/multus-additional-cni-plugins-sld25"
Jul 17 12:29:52.992476 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.991873    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/688df482-0ac3-4a72-9632-c6a8c18b0b8d-tmp-dir\") pod \"node-resolver-g8rzg\" (UID: \"688df482-0ac3-4a72-9632-c6a8c18b0b8d\") " pod="openshift-dns/node-resolver-g8rzg"
Jul 17 12:29:52.993026 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.993003    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/146d6cf8-52ca-4482-9abb-ef40e770eb13-agent-certs\") pod \"konnectivity-agent-5v7sc\" (UID: \"146d6cf8-52ca-4482-9abb-ef40e770eb13\") " pod="kube-system/konnectivity-agent-5v7sc"
Jul 17 12:29:52.993079 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:52.993063    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ab622df-86d3-4cc3-aba7-123f4bc0c996-ovn-node-metrics-cert\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:53.011197 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:53.011159    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn2b9\" (UniqueName: \"kubernetes.io/projected/7ab622df-86d3-4cc3-aba7-123f4bc0c996-kube-api-access-dn2b9\") pod \"ovnkube-node-9f958\" (UID: \"7ab622df-86d3-4cc3-aba7-123f4bc0c996\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:53.011831 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:53.011804    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntrg7\" (UniqueName: \"kubernetes.io/projected/937e7356-528f-4a56-a80e-9486657ec301-kube-api-access-ntrg7\") pod \"network-metrics-daemon-lxrxw\" (UID: \"937e7356-528f-4a56-a80e-9486657ec301\") " pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:29:53.012060 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:53.012041    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 12:29:53.012132 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:53.012067    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 12:29:53.012132 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:53.012081    2564 projected.go:196] Error preparing data for projected volume kube-api-access-pmwjl for pod openshift-network-diagnostics/network-check-target-xl2np: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:29:53.012222 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:53.012210    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a72fcab3-1981-4637-a041-225335390f1d-kube-api-access-pmwjl podName:a72fcab3-1981-4637-a041-225335390f1d nodeName:}" failed. No retries permitted until 2026-07-17 12:29:53.512179157 +0000 UTC m=+3.092291156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pmwjl" (UniqueName: "kubernetes.io/projected/a72fcab3-1981-4637-a041-225335390f1d-kube-api-access-pmwjl") pod "network-check-target-xl2np" (UID: "a72fcab3-1981-4637-a041-225335390f1d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:29:53.014634 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:53.014601    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z8tx\" (UniqueName: \"kubernetes.io/projected/cc19f981-8855-47ce-98c7-6e8e9c0e65d2-kube-api-access-4z8tx\") pod \"node-ca-xxld5\" (UID: \"cc19f981-8855-47ce-98c7-6e8e9c0e65d2\") " pod="openshift-image-registry/node-ca-xxld5"
Jul 17 12:29:53.014750 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:53.014679    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjgdr\" (UniqueName: \"kubernetes.io/projected/b09e01d5-4dca-4a57-903d-f6daacfdb5d9-kube-api-access-wjgdr\") pod \"multus-additional-cni-plugins-sld25\" (UID: \"b09e01d5-4dca-4a57-903d-f6daacfdb5d9\") " pod="openshift-multus/multus-additional-cni-plugins-sld25"
Jul 17 12:29:53.014964 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:53.014947    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7gd8\" (UniqueName: \"kubernetes.io/projected/688df482-0ac3-4a72-9632-c6a8c18b0b8d-kube-api-access-r7gd8\") pod \"node-resolver-g8rzg\" (UID: \"688df482-0ac3-4a72-9632-c6a8c18b0b8d\") " pod="openshift-dns/node-resolver-g8rzg"
Jul 17 12:29:53.087830 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:53.087791    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8cbb35a0cc4730bb524456da0fe6cdf.slice/crio-ab4813f512f939cbc9ce0fcb9c165db52dda6e15d31a606f9a9d927129429db8 WatchSource:0}: Error finding container ab4813f512f939cbc9ce0fcb9c165db52dda6e15d31a606f9a9d927129429db8: Status 404 returned error can't find the container with id ab4813f512f939cbc9ce0fcb9c165db52dda6e15d31a606f9a9d927129429db8
Jul 17 12:29:53.093231 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:53.093210    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 12:29:53.098441 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:53.098411    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7ql7p"
Jul 17 12:29:53.100491 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:53.100467    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf283087a6d8d1084ff25d034c2bbc6d1.slice/crio-dec2baedd8ca68fe4b6b09dc28c21c2398cde522cccb3784b673bff0e1898403 WatchSource:0}: Error finding container dec2baedd8ca68fe4b6b09dc28c21c2398cde522cccb3784b673bff0e1898403: Status 404 returned error can't find the container with id dec2baedd8ca68fe4b6b09dc28c21c2398cde522cccb3784b673bff0e1898403
Jul 17 12:29:53.107472 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:53.107448    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc47b98e9_a381_4815_b8d9_4737cf123e0d.slice/crio-9f40852854519f580e974768c1e5ab8166646cae9c7a5489dffc3554886f5133 WatchSource:0}: Error finding container 9f40852854519f580e974768c1e5ab8166646cae9c7a5489dffc3554886f5133: Status 404 returned error can't find the container with id 9f40852854519f580e974768c1e5ab8166646cae9c7a5489dffc3554886f5133
Jul 17 12:29:53.110704 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:53.110684    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsvdx"
Jul 17 12:29:53.119696 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:53.119673    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83656ae5_efe4_426a_9a4c_080b18541deb.slice/crio-da70e4a6137716c15205fe36be64b9616011f218e31e2b1f7f3f710579bd10f0 WatchSource:0}: Error finding container da70e4a6137716c15205fe36be64b9616011f218e31e2b1f7f3f710579bd10f0: Status 404 returned error can't find the container with id da70e4a6137716c15205fe36be64b9616011f218e31e2b1f7f3f710579bd10f0
Jul 17 12:29:53.130033 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:53.130010    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xdqkr"
Jul 17 12:29:53.138649 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:53.138623    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe8942fd_34dd_4b5b_ba05_0c6488374820.slice/crio-5c81fa808f9072a2eede49d773b94bc8155f77ffbd115a90c007f999f022d3fe WatchSource:0}: Error finding container 5c81fa808f9072a2eede49d773b94bc8155f77ffbd115a90c007f999f022d3fe: Status 404 returned error can't find the container with id 5c81fa808f9072a2eede49d773b94bc8155f77ffbd115a90c007f999f022d3fe
Jul 17 12:29:53.143554 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:53.143535    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vtsx2"
Jul 17 12:29:53.153095 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:53.153072    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf92fb403_4621_413f_9e6b_c041411f774a.slice/crio-5f81eb21c70138ea2b022c029028dabf51c01a35723874873927a33a0e674cc9 WatchSource:0}: Error finding container 5f81eb21c70138ea2b022c029028dabf51c01a35723874873927a33a0e674cc9: Status 404 returned error can't find the container with id 5f81eb21c70138ea2b022c029028dabf51c01a35723874873927a33a0e674cc9
Jul 17 12:29:53.159184 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:53.159168    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5v7sc"
Jul 17 12:29:53.167832 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:53.167790    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146d6cf8_52ca_4482_9abb_ef40e770eb13.slice/crio-efefb3ca2221dc53234644ec5681121e7d6263023a96b0ac521ddc3c6ef2767a WatchSource:0}: Error finding container efefb3ca2221dc53234644ec5681121e7d6263023a96b0ac521ddc3c6ef2767a: Status 404 returned error can't find the container with id efefb3ca2221dc53234644ec5681121e7d6263023a96b0ac521ddc3c6ef2767a
Jul 17 12:29:53.179549 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:53.179524    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xxld5"
Jul 17 12:29:53.188395 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:53.188373    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc19f981_8855_47ce_98c7_6e8e9c0e65d2.slice/crio-3c43618f87b78fb1312742fb8023d464c76ee5ad932dc27cc144a1a620057a9b WatchSource:0}: Error finding container 3c43618f87b78fb1312742fb8023d464c76ee5ad932dc27cc144a1a620057a9b: Status 404 returned error can't find the container with id 3c43618f87b78fb1312742fb8023d464c76ee5ad932dc27cc144a1a620057a9b
Jul 17 12:29:53.194033 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:53.194012    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sld25"
Jul 17 12:29:53.203041 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:53.203015    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb09e01d5_4dca_4a57_903d_f6daacfdb5d9.slice/crio-bf7586046f2081e51ad3e7810249580a05ecb65e1e96d31c22152234ee7665fc WatchSource:0}: Error finding container bf7586046f2081e51ad3e7810249580a05ecb65e1e96d31c22152234ee7665fc: Status 404 returned error can't find the container with id bf7586046f2081e51ad3e7810249580a05ecb65e1e96d31c22152234ee7665fc
Jul 17 12:29:53.216052 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:53.216033    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g8rzg"
Jul 17 12:29:53.221084 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:53.221061    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:29:53.230449 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:53.230421    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688df482_0ac3_4a72_9632_c6a8c18b0b8d.slice/crio-a7c9bd003107199fe11d4573953efccd84750f8bde2aedf2428a2bcea55a7045 WatchSource:0}: Error finding container a7c9bd003107199fe11d4573953efccd84750f8bde2aedf2428a2bcea55a7045: Status 404 returned error can't find the container with id a7c9bd003107199fe11d4573953efccd84750f8bde2aedf2428a2bcea55a7045
Jul 17 12:29:53.231188 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:29:53.231165    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab622df_86d3_4cc3_aba7_123f4bc0c996.slice/crio-4d7125e00129f200c9c339c7c172ee2a01f42df72b5a1817f647af21f98fd19e WatchSource:0}: Error finding container 4d7125e00129f200c9c339c7c172ee2a01f42df72b5a1817f647af21f98fd19e: Status 404 returned error can't find the container with id 4d7125e00129f200c9c339c7c172ee2a01f42df72b5a1817f647af21f98fd19e
Jul 17 12:29:53.233143 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:53.233120    2564 kuberuntime_manager.go:1459] "Unhandled Error" err="container ovn-controller start failed in pod ovnkube-node-9f958_openshift-ovn-kubernetes(7ab622df-86d3-4cc3-aba7-123f4bc0c996): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Jul 17 12:29:53.234503 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:53.234482    2564 kuberuntime_manager.go:1459] "Unhandled Error" err="container kube-rbac-proxy-node start failed in pod ovnkube-node-9f958_openshift-ovn-kubernetes(7ab622df-86d3-4cc3-aba7-123f4bc0c996): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Jul 17 12:29:53.237790 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:53.237704    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:241e47351d3f7eef8ef00c2c9acba53c54ac799f4e751de5b4d4e13016eaa997\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:534ab2ea2f6c22eaf4d2267274394590a29319c3d5e0da72bef87bc86f428dec\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:241e47351d3f7eef8ef00c2c9acba53c54ac799f4e751de5b4d4e13016eaa997\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:241e47351d3f7eef8ef00c2c9acba53c54ac799f4e751de5b4d4e13016eaa997\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:241e47351d3f7eef8ef00c2c9acba53c54ac799f4e751de5b4d4e13016eaa997\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:241e47351d3f7eef8ef00c2c9acba53c54ac799f4e751de5b4d4e13016eaa997\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-9f958" podUID="7ab622df-86d3-4cc3-aba7-123f4bc0c996"
Jul 17 12:29:53.494884 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:53.494760    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/937e7356-528f-4a56-a80e-9486657ec301-metrics-certs\") pod \"network-metrics-daemon-lxrxw\" (UID: \"937e7356-528f-4a56-a80e-9486657ec301\") " pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:29:53.495048 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:53.494916    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c803ba62-1a8b-4b71-a355-fc4427be817b-original-pull-secret\") pod \"global-pull-secret-syncer-9b4b4\" (UID: \"c803ba62-1a8b-4b71-a355-fc4427be817b\") " pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:29:53.495048 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:53.495035    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 12:29:53.495048 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:53.495035    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:29:53.495175 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:53.495098    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c803ba62-1a8b-4b71-a355-fc4427be817b-original-pull-secret podName:c803ba62-1a8b-4b71-a355-fc4427be817b nodeName:}" failed. No retries permitted until 2026-07-17 12:29:54.49508303 +0000 UTC m=+4.075195033 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c803ba62-1a8b-4b71-a355-fc4427be817b-original-pull-secret") pod "global-pull-secret-syncer-9b4b4" (UID: "c803ba62-1a8b-4b71-a355-fc4427be817b") : object "kube-system"/"original-pull-secret" not registered
Jul 17 12:29:53.495175 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:53.495116    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/937e7356-528f-4a56-a80e-9486657ec301-metrics-certs podName:937e7356-528f-4a56-a80e-9486657ec301 nodeName:}" failed. No retries permitted until 2026-07-17 12:29:54.495107107 +0000 UTC m=+4.075219105 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/937e7356-528f-4a56-a80e-9486657ec301-metrics-certs") pod "network-metrics-daemon-lxrxw" (UID: "937e7356-528f-4a56-a80e-9486657ec301") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:29:53.595553 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:53.595509    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmwjl\" (UniqueName: \"kubernetes.io/projected/a72fcab3-1981-4637-a041-225335390f1d-kube-api-access-pmwjl\") pod \"network-check-target-xl2np\" (UID: \"a72fcab3-1981-4637-a041-225335390f1d\") " pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:29:53.595743 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:53.595706    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 12:29:53.595743 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:53.595726    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 12:29:53.595888 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:53.595758    2564 projected.go:196] Error preparing data for projected volume kube-api-access-pmwjl for pod openshift-network-diagnostics/network-check-target-xl2np: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:29:53.595888 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:53.595822    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a72fcab3-1981-4637-a041-225335390f1d-kube-api-access-pmwjl podName:a72fcab3-1981-4637-a041-225335390f1d nodeName:}" failed. No retries permitted until 2026-07-17 12:29:54.5958037 +0000 UTC m=+4.175915702 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pmwjl" (UniqueName: "kubernetes.io/projected/a72fcab3-1981-4637-a041-225335390f1d-kube-api-access-pmwjl") pod "network-check-target-xl2np" (UID: "a72fcab3-1981-4637-a041-225335390f1d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:29:53.816241 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:53.816048    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 12:24:53 +0000 UTC" deadline="2028-01-30 00:06:45.285670857 +0000 UTC"
Jul 17 12:29:53.816241 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:53.816122    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13475h36m51.469553256s"
Jul 17 12:29:54.008854 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:54.008823    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:29:54.009716 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:54.008974    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b4b4" podUID="c803ba62-1a8b-4b71-a355-fc4427be817b"
Jul 17 12:29:54.009716 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:54.009398    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:29:54.009716 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:54.009497    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xl2np" podUID="a72fcab3-1981-4637-a041-225335390f1d"
Jul 17 12:29:54.026276 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:54.026211    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9f958" event={"ID":"7ab622df-86d3-4cc3-aba7-123f4bc0c996","Type":"ContainerStarted","Data":"4d7125e00129f200c9c339c7c172ee2a01f42df72b5a1817f647af21f98fd19e"}
Jul 17 12:29:54.038596 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:54.038493    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sld25" event={"ID":"b09e01d5-4dca-4a57-903d-f6daacfdb5d9","Type":"ContainerStarted","Data":"bf7586046f2081e51ad3e7810249580a05ecb65e1e96d31c22152234ee7665fc"}
Jul 17 12:29:54.053029 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:54.052993    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5v7sc" event={"ID":"146d6cf8-52ca-4482-9abb-ef40e770eb13","Type":"ContainerStarted","Data":"efefb3ca2221dc53234644ec5681121e7d6263023a96b0ac521ddc3c6ef2767a"}
Jul 17 12:29:54.058311 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:54.058234    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:241e47351d3f7eef8ef00c2c9acba53c54ac799f4e751de5b4d4e13016eaa997\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:241e47351d3f7eef8ef00c2c9acba53c54ac799f4e751de5b4d4e13016eaa997\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:534ab2ea2f6c22eaf4d2267274394590a29319c3d5e0da72bef87bc86f428dec\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:534ab2ea2f6c22eaf4d2267274394590a29319c3d5e0da72bef87bc86f428dec\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:241e47351d3f7eef8ef00c2c9acba53c54ac799f4e751de5b4d4e13016eaa997\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:241e47351d3f7eef8ef00c2c9acba53c54ac799f4e751de5b4d4e13016eaa997\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:241e47351d3f7eef8ef00c2c9acba53c54ac799f4e751de5b4d4e13016eaa997\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:241e47351d3f7eef8ef00c2c9acba53c54ac799f4e751de5b4d4e13016eaa997\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-9f958" podUID="7ab622df-86d3-4cc3-aba7-123f4bc0c996"
Jul 17 12:29:54.068075 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:54.067990    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vtsx2" event={"ID":"f92fb403-4621-413f-9e6b-c041411f774a","Type":"ContainerStarted","Data":"5f81eb21c70138ea2b022c029028dabf51c01a35723874873927a33a0e674cc9"}
Jul 17 12:29:54.078633 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:54.078593    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xdqkr" event={"ID":"fe8942fd-34dd-4b5b-ba05-0c6488374820","Type":"ContainerStarted","Data":"5c81fa808f9072a2eede49d773b94bc8155f77ffbd115a90c007f999f022d3fe"}
Jul 17 12:29:54.099424 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:54.099386    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsvdx" event={"ID":"83656ae5-efe4-426a-9a4c-080b18541deb","Type":"ContainerStarted","Data":"da70e4a6137716c15205fe36be64b9616011f218e31e2b1f7f3f710579bd10f0"}
Jul 17 12:29:54.111598 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:54.111564    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7ql7p" event={"ID":"c47b98e9-a381-4815-b8d9-4737cf123e0d","Type":"ContainerStarted","Data":"9f40852854519f580e974768c1e5ab8166646cae9c7a5489dffc3554886f5133"}
Jul 17 12:29:54.127857 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:54.127819    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g8rzg" event={"ID":"688df482-0ac3-4a72-9632-c6a8c18b0b8d","Type":"ContainerStarted","Data":"a7c9bd003107199fe11d4573953efccd84750f8bde2aedf2428a2bcea55a7045"}
Jul 17 12:29:54.155855 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:54.155811    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xxld5" event={"ID":"cc19f981-8855-47ce-98c7-6e8e9c0e65d2","Type":"ContainerStarted","Data":"3c43618f87b78fb1312742fb8023d464c76ee5ad932dc27cc144a1a620057a9b"}
Jul 17 12:29:54.165617 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:54.165573    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-99.ec2.internal" event={"ID":"f283087a6d8d1084ff25d034c2bbc6d1","Type":"ContainerStarted","Data":"dec2baedd8ca68fe4b6b09dc28c21c2398cde522cccb3784b673bff0e1898403"}
Jul 17 12:29:54.180582 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:54.180527    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-99.ec2.internal" event={"ID":"c8cbb35a0cc4730bb524456da0fe6cdf","Type":"ContainerStarted","Data":"ab4813f512f939cbc9ce0fcb9c165db52dda6e15d31a606f9a9d927129429db8"}
Jul 17 12:29:54.506796 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:54.506733    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c803ba62-1a8b-4b71-a355-fc4427be817b-original-pull-secret\") pod \"global-pull-secret-syncer-9b4b4\" (UID: \"c803ba62-1a8b-4b71-a355-fc4427be817b\") " pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:29:54.506992 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:54.506848    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/937e7356-528f-4a56-a80e-9486657ec301-metrics-certs\") pod \"network-metrics-daemon-lxrxw\" (UID: \"937e7356-528f-4a56-a80e-9486657ec301\") " pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:29:54.507056 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:54.507004    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:29:54.507110 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:54.507075    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/937e7356-528f-4a56-a80e-9486657ec301-metrics-certs podName:937e7356-528f-4a56-a80e-9486657ec301 nodeName:}" failed. No retries permitted until 2026-07-17 12:29:56.507054688 +0000 UTC m=+6.087166690 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/937e7356-528f-4a56-a80e-9486657ec301-metrics-certs") pod "network-metrics-daemon-lxrxw" (UID: "937e7356-528f-4a56-a80e-9486657ec301") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:29:54.507110 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:54.507005    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 12:29:54.507231 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:54.507173    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c803ba62-1a8b-4b71-a355-fc4427be817b-original-pull-secret podName:c803ba62-1a8b-4b71-a355-fc4427be817b nodeName:}" failed. No retries permitted until 2026-07-17 12:29:56.507150183 +0000 UTC m=+6.087262207 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c803ba62-1a8b-4b71-a355-fc4427be817b-original-pull-secret") pod "global-pull-secret-syncer-9b4b4" (UID: "c803ba62-1a8b-4b71-a355-fc4427be817b") : object "kube-system"/"original-pull-secret" not registered
Jul 17 12:29:54.608171 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:54.608130    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmwjl\" (UniqueName: \"kubernetes.io/projected/a72fcab3-1981-4637-a041-225335390f1d-kube-api-access-pmwjl\") pod \"network-check-target-xl2np\" (UID: \"a72fcab3-1981-4637-a041-225335390f1d\") " pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:29:54.608349 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:54.608332    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 12:29:54.608426 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:54.608357    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 12:29:54.608426 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:54.608371    2564 projected.go:196] Error preparing data for projected volume kube-api-access-pmwjl for pod openshift-network-diagnostics/network-check-target-xl2np: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:29:54.608553 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:54.608433    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a72fcab3-1981-4637-a041-225335390f1d-kube-api-access-pmwjl podName:a72fcab3-1981-4637-a041-225335390f1d nodeName:}" failed. No retries permitted until 2026-07-17 12:29:56.60841435 +0000 UTC m=+6.188526364 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pmwjl" (UniqueName: "kubernetes.io/projected/a72fcab3-1981-4637-a041-225335390f1d-kube-api-access-pmwjl") pod "network-check-target-xl2np" (UID: "a72fcab3-1981-4637-a041-225335390f1d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:29:54.817350 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:54.817250    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 12:24:53 +0000 UTC" deadline="2028-03-22 06:48:33.753529858 +0000 UTC"
Jul 17 12:29:54.817350 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:54.817297    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14730h18m38.936238568s"
Jul 17 12:29:55.011846 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:55.011810    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:29:55.012483 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:55.011959    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxrxw" podUID="937e7356-528f-4a56-a80e-9486657ec301"
Jul 17 12:29:56.009183 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:56.009108    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:29:56.009383 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:56.009284    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xl2np" podUID="a72fcab3-1981-4637-a041-225335390f1d"
Jul 17 12:29:56.009816 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:56.009794    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:29:56.009927 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:56.009894    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b4b4" podUID="c803ba62-1a8b-4b71-a355-fc4427be817b"
Jul 17 12:29:56.525343 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:56.525297    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c803ba62-1a8b-4b71-a355-fc4427be817b-original-pull-secret\") pod \"global-pull-secret-syncer-9b4b4\" (UID: \"c803ba62-1a8b-4b71-a355-fc4427be817b\") " pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:29:56.525934 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:56.525438    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/937e7356-528f-4a56-a80e-9486657ec301-metrics-certs\") pod \"network-metrics-daemon-lxrxw\" (UID: \"937e7356-528f-4a56-a80e-9486657ec301\") " pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:29:56.525934 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:56.525505    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 12:29:56.525934 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:56.525555    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:29:56.525934 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:56.525593    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c803ba62-1a8b-4b71-a355-fc4427be817b-original-pull-secret podName:c803ba62-1a8b-4b71-a355-fc4427be817b nodeName:}" failed. No retries permitted until 2026-07-17 12:30:00.525571014 +0000 UTC m=+10.105683039 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c803ba62-1a8b-4b71-a355-fc4427be817b-original-pull-secret") pod "global-pull-secret-syncer-9b4b4" (UID: "c803ba62-1a8b-4b71-a355-fc4427be817b") : object "kube-system"/"original-pull-secret" not registered
Jul 17 12:29:56.525934 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:56.525622    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/937e7356-528f-4a56-a80e-9486657ec301-metrics-certs podName:937e7356-528f-4a56-a80e-9486657ec301 nodeName:}" failed. No retries permitted until 2026-07-17 12:30:00.525603988 +0000 UTC m=+10.105716002 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/937e7356-528f-4a56-a80e-9486657ec301-metrics-certs") pod "network-metrics-daemon-lxrxw" (UID: "937e7356-528f-4a56-a80e-9486657ec301") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:29:56.626240 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:56.626200    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmwjl\" (UniqueName: \"kubernetes.io/projected/a72fcab3-1981-4637-a041-225335390f1d-kube-api-access-pmwjl\") pod \"network-check-target-xl2np\" (UID: \"a72fcab3-1981-4637-a041-225335390f1d\") " pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:29:56.626893 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:56.626421    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 12:29:56.626893 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:56.626453    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 12:29:56.626893 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:56.626468    2564 projected.go:196] Error preparing data for projected volume kube-api-access-pmwjl for pod openshift-network-diagnostics/network-check-target-xl2np: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:29:56.626893 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:56.626535    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a72fcab3-1981-4637-a041-225335390f1d-kube-api-access-pmwjl podName:a72fcab3-1981-4637-a041-225335390f1d nodeName:}" failed. No retries permitted until 2026-07-17 12:30:00.62651446 +0000 UTC m=+10.206626475 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pmwjl" (UniqueName: "kubernetes.io/projected/a72fcab3-1981-4637-a041-225335390f1d-kube-api-access-pmwjl") pod "network-check-target-xl2np" (UID: "a72fcab3-1981-4637-a041-225335390f1d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:29:57.009925 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:57.009832    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:29:57.010100 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:57.010004    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxrxw" podUID="937e7356-528f-4a56-a80e-9486657ec301"
Jul 17 12:29:58.008826 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:58.008477    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:29:58.008826 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:58.008535    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:29:58.008826 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:58.008637    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xl2np" podUID="a72fcab3-1981-4637-a041-225335390f1d"
Jul 17 12:29:58.009410 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:58.009098    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b4b4" podUID="c803ba62-1a8b-4b71-a355-fc4427be817b"
Jul 17 12:29:59.009045 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:29:59.009006    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:29:59.009533 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:29:59.009178    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxrxw" podUID="937e7356-528f-4a56-a80e-9486657ec301"
Jul 17 12:30:00.008962 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:00.008929    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:30:00.009150 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:00.008932    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:30:00.009150 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:00.009070    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xl2np" podUID="a72fcab3-1981-4637-a041-225335390f1d"
Jul 17 12:30:00.009150 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:00.009132    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b4b4" podUID="c803ba62-1a8b-4b71-a355-fc4427be817b"
Jul 17 12:30:00.560315 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:00.560012    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c803ba62-1a8b-4b71-a355-fc4427be817b-original-pull-secret\") pod \"global-pull-secret-syncer-9b4b4\" (UID: \"c803ba62-1a8b-4b71-a355-fc4427be817b\") " pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:30:00.560315 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:00.560148    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/937e7356-528f-4a56-a80e-9486657ec301-metrics-certs\") pod \"network-metrics-daemon-lxrxw\" (UID: \"937e7356-528f-4a56-a80e-9486657ec301\") " pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:30:00.560315 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:00.560281    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 12:30:00.560643 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:00.560282    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:30:00.560643 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:00.560438    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c803ba62-1a8b-4b71-a355-fc4427be817b-original-pull-secret podName:c803ba62-1a8b-4b71-a355-fc4427be817b nodeName:}" failed. No retries permitted until 2026-07-17 12:30:08.560413782 +0000 UTC m=+18.140525791 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c803ba62-1a8b-4b71-a355-fc4427be817b-original-pull-secret") pod "global-pull-secret-syncer-9b4b4" (UID: "c803ba62-1a8b-4b71-a355-fc4427be817b") : object "kube-system"/"original-pull-secret" not registered
Jul 17 12:30:00.560643 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:00.560466    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/937e7356-528f-4a56-a80e-9486657ec301-metrics-certs podName:937e7356-528f-4a56-a80e-9486657ec301 nodeName:}" failed. No retries permitted until 2026-07-17 12:30:08.560452596 +0000 UTC m=+18.140564604 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/937e7356-528f-4a56-a80e-9486657ec301-metrics-certs") pod "network-metrics-daemon-lxrxw" (UID: "937e7356-528f-4a56-a80e-9486657ec301") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:30:00.660514 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:00.660470    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmwjl\" (UniqueName: \"kubernetes.io/projected/a72fcab3-1981-4637-a041-225335390f1d-kube-api-access-pmwjl\") pod \"network-check-target-xl2np\" (UID: \"a72fcab3-1981-4637-a041-225335390f1d\") " pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:30:00.660690 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:00.660648    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 12:30:00.660690 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:00.660673    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 12:30:00.660690 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:00.660688    2564 projected.go:196] Error preparing data for projected volume kube-api-access-pmwjl for pod openshift-network-diagnostics/network-check-target-xl2np: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:30:00.660847 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:00.660779    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a72fcab3-1981-4637-a041-225335390f1d-kube-api-access-pmwjl podName:a72fcab3-1981-4637-a041-225335390f1d nodeName:}" failed. No retries permitted until 2026-07-17 12:30:08.660763538 +0000 UTC m=+18.240875552 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pmwjl" (UniqueName: "kubernetes.io/projected/a72fcab3-1981-4637-a041-225335390f1d-kube-api-access-pmwjl") pod "network-check-target-xl2np" (UID: "a72fcab3-1981-4637-a041-225335390f1d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:30:01.009845 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:01.009753    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:30:01.010315 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:01.009916    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxrxw" podUID="937e7356-528f-4a56-a80e-9486657ec301"
Jul 17 12:30:02.009341 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:02.009304    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:30:02.009538 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:02.009357    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:30:02.009538 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:02.009461    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b4b4" podUID="c803ba62-1a8b-4b71-a355-fc4427be817b"
Jul 17 12:30:02.009671 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:02.009627    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xl2np" podUID="a72fcab3-1981-4637-a041-225335390f1d"
Jul 17 12:30:03.009248 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:03.009208    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:30:03.009764 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:03.009346    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxrxw" podUID="937e7356-528f-4a56-a80e-9486657ec301"
Jul 17 12:30:04.008685 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:04.008651    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:30:04.008685 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:04.008685    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:30:04.008954 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:04.008794    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b4b4" podUID="c803ba62-1a8b-4b71-a355-fc4427be817b"
Jul 17 12:30:04.008954 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:04.008932    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xl2np" podUID="a72fcab3-1981-4637-a041-225335390f1d"
Jul 17 12:30:05.008827 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:05.008784    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:30:05.009264 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:05.009067    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxrxw" podUID="937e7356-528f-4a56-a80e-9486657ec301"
Jul 17 12:30:06.009142 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:06.009113    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:30:06.009635 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:06.009246    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b4b4" podUID="c803ba62-1a8b-4b71-a355-fc4427be817b"
Jul 17 12:30:06.009635 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:06.009297    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:30:06.009635 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:06.009382    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xl2np" podUID="a72fcab3-1981-4637-a041-225335390f1d"
Jul 17 12:30:07.010447 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:07.010411    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:30:07.010919 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:07.010562    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxrxw" podUID="937e7356-528f-4a56-a80e-9486657ec301"
Jul 17 12:30:07.218591 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:07.218499    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vtsx2" event={"ID":"f92fb403-4621-413f-9e6b-c041411f774a","Type":"ContainerStarted","Data":"98e105976bf5ef558915a06b5acccf268bc0a46b5409feddc1b5595a39752194"}
Jul 17 12:30:07.220585 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:07.220501    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7ql7p" event={"ID":"c47b98e9-a381-4815-b8d9-4737cf123e0d","Type":"ContainerStarted","Data":"2e640da79adb3aba1aa9fbf0200cb74a68dc18af31f61895ed0d415811b50141"}
Jul 17 12:30:07.222447 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:07.222411    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-99.ec2.internal" event={"ID":"f283087a6d8d1084ff25d034c2bbc6d1","Type":"ContainerStarted","Data":"e808af54d179b81b0b74e96328b382aac14b40222f99d6fd7d10a22309f42eb0"}
Jul 17 12:30:07.269954 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:07.269902    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-99.ec2.internal" podStartSLOduration=15.269887929 podStartE2EDuration="15.269887929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:30:07.269776571 +0000 UTC m=+16.849888588" watchObservedRunningTime="2026-07-17 12:30:07.269887929 +0000 UTC m=+16.849999947"
Jul 17 12:30:07.270276 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:07.270256    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vtsx2" podStartSLOduration=2.7512789460000002 podStartE2EDuration="16.270250451s" totalImagesPullingTime="13.518971505s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:29:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:30:07.248183974 +0000 UTC m=+16.828295994" watchObservedRunningTime="2026-07-17 12:30:07.270250451 +0000 UTC m=+16.850362470"
Jul 17 12:30:07.300274 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:07.300226    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7ql7p" podStartSLOduration=2.524891437 podStartE2EDuration="16.300209523s" totalImagesPullingTime="13.775318086s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:29:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:30:07.300000514 +0000 UTC m=+16.880112532" watchObservedRunningTime="2026-07-17 12:30:07.300209523 +0000 UTC m=+16.880321541"
Jul 17 12:30:08.009404 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:08.009369    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:30:08.009600 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:08.009380    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:30:08.009600 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:08.009517    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b4b4" podUID="c803ba62-1a8b-4b71-a355-fc4427be817b"
Jul 17 12:30:08.009600 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:08.009547    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xl2np" podUID="a72fcab3-1981-4637-a041-225335390f1d"
Jul 17 12:30:08.226593 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:08.226301    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g8rzg" event={"ID":"688df482-0ac3-4a72-9632-c6a8c18b0b8d","Type":"ContainerStarted","Data":"c0b978b32064120be3586b3eed9c2a81012bb76c4d72cdf9e6fbd6684c8776e0"}
Jul 17 12:30:08.228502 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:08.228456    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xxld5" event={"ID":"cc19f981-8855-47ce-98c7-6e8e9c0e65d2","Type":"ContainerStarted","Data":"27f323f5a03c3d23049379ccabde413fdf682e8a922bbdf6b097040f8df1feb7"}
Jul 17 12:30:08.230406 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:08.230368    2564 generic.go:356] "Generic (PLEG): container finished" podID="c8cbb35a0cc4730bb524456da0fe6cdf" containerID="7e66e6b1eac383c7d4d4fb2c4322cb4034669ffeff049ea14003bde6595916e7" exitCode=0
Jul 17 12:30:08.230923 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:08.230891    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-99.ec2.internal" event={"ID":"c8cbb35a0cc4730bb524456da0fe6cdf","Type":"ContainerDied","Data":"7e66e6b1eac383c7d4d4fb2c4322cb4034669ffeff049ea14003bde6595916e7"}
Jul 17 12:30:08.233738 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:08.233691    2564 generic.go:356] "Generic (PLEG): container finished" podID="b09e01d5-4dca-4a57-903d-f6daacfdb5d9" containerID="1ae9b59d4339e0a3673e593c93adef36512d4635b75daefa6db5aab1b7e0a54e" exitCode=0
Jul 17 12:30:08.234328 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:08.233878    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sld25" event={"ID":"b09e01d5-4dca-4a57-903d-f6daacfdb5d9","Type":"ContainerDied","Data":"1ae9b59d4339e0a3673e593c93adef36512d4635b75daefa6db5aab1b7e0a54e"}
Jul 17 12:30:08.236214 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:08.236191    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5v7sc" event={"ID":"146d6cf8-52ca-4482-9abb-ef40e770eb13","Type":"ContainerStarted","Data":"f89bf3a910e8caebf5b5595667edd54916589d1f91c697e1a09706c67828078d"}
Jul 17 12:30:08.238784 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:08.238747    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsvdx" event={"ID":"83656ae5-efe4-426a-9a4c-080b18541deb","Type":"ContainerStarted","Data":"3be71fdb37f7145383c4a51b75b1b163b56f3421bc9610cc6a8586799a3617dc"}
Jul 17 12:30:08.244055 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:08.244010    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-g8rzg" podStartSLOduration=2.815914609 podStartE2EDuration="16.243992391s" totalImagesPullingTime="13.428077782s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:29:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:30:08.2435526 +0000 UTC m=+17.823664619" watchObservedRunningTime="2026-07-17 12:30:08.243992391 +0000 UTC m=+17.824104412"
Jul 17 12:30:08.319252 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:08.319182    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5v7sc" podStartSLOduration=3.829182127 podStartE2EDuration="17.319161005s" totalImagesPullingTime="13.489978878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:29:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:30:08.316879903 +0000 UTC m=+17.896991921" watchObservedRunningTime="2026-07-17 12:30:08.319161005 +0000 UTC m=+17.899273024"
Jul 17 12:30:08.340201 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:08.340129    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xxld5" podStartSLOduration=2.869553811 podStartE2EDuration="16.340112658s" totalImagesPullingTime="13.470558847s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:29:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:30:08.339450525 +0000 UTC m=+17.919562545" watchObservedRunningTime="2026-07-17 12:30:08.340112658 +0000 UTC m=+17.920224676"
Jul 17 12:30:08.627463 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:08.627367    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c803ba62-1a8b-4b71-a355-fc4427be817b-original-pull-secret\") pod \"global-pull-secret-syncer-9b4b4\" (UID: \"c803ba62-1a8b-4b71-a355-fc4427be817b\") " pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:30:08.627463 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:08.627450    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/937e7356-528f-4a56-a80e-9486657ec301-metrics-certs\") pod \"network-metrics-daemon-lxrxw\" (UID: \"937e7356-528f-4a56-a80e-9486657ec301\") " pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:30:08.627688 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:08.627555    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 12:30:08.627688 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:08.627595    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:30:08.627688 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:08.627643    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c803ba62-1a8b-4b71-a355-fc4427be817b-original-pull-secret podName:c803ba62-1a8b-4b71-a355-fc4427be817b nodeName:}" failed. No retries permitted until 2026-07-17 12:30:24.627619117 +0000 UTC m=+34.207731132 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c803ba62-1a8b-4b71-a355-fc4427be817b-original-pull-secret") pod "global-pull-secret-syncer-9b4b4" (UID: "c803ba62-1a8b-4b71-a355-fc4427be817b") : object "kube-system"/"original-pull-secret" not registered
Jul 17 12:30:08.627688 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:08.627662    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/937e7356-528f-4a56-a80e-9486657ec301-metrics-certs podName:937e7356-528f-4a56-a80e-9486657ec301 nodeName:}" failed. No retries permitted until 2026-07-17 12:30:24.6276542 +0000 UTC m=+34.207766196 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/937e7356-528f-4a56-a80e-9486657ec301-metrics-certs") pod "network-metrics-daemon-lxrxw" (UID: "937e7356-528f-4a56-a80e-9486657ec301") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:30:08.727941 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:08.727909    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmwjl\" (UniqueName: \"kubernetes.io/projected/a72fcab3-1981-4637-a041-225335390f1d-kube-api-access-pmwjl\") pod \"network-check-target-xl2np\" (UID: \"a72fcab3-1981-4637-a041-225335390f1d\") " pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:30:08.728128 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:08.728106    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 12:30:08.728170 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:08.728137    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 12:30:08.728170 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:08.728152    2564 projected.go:196] Error preparing data for projected volume kube-api-access-pmwjl for pod openshift-network-diagnostics/network-check-target-xl2np: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:30:08.728265 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:08.728221    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a72fcab3-1981-4637-a041-225335390f1d-kube-api-access-pmwjl podName:a72fcab3-1981-4637-a041-225335390f1d nodeName:}" failed. No retries permitted until 2026-07-17 12:30:24.728202555 +0000 UTC m=+34.308314564 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pmwjl" (UniqueName: "kubernetes.io/projected/a72fcab3-1981-4637-a041-225335390f1d-kube-api-access-pmwjl") pod "network-check-target-xl2np" (UID: "a72fcab3-1981-4637-a041-225335390f1d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:30:09.009600 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:09.009513    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:30:09.009775 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:09.009640    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxrxw" podUID="937e7356-528f-4a56-a80e-9486657ec301"
Jul 17 12:30:09.243762 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:09.243692    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-99.ec2.internal" event={"ID":"c8cbb35a0cc4730bb524456da0fe6cdf","Type":"ContainerStarted","Data":"6de2c10888ea65ee3dda68490c528fd745431e83574eac19d7687a4ad8ac9673"}
Jul 17 12:30:09.246349 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:09.246320    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xdqkr" event={"ID":"fe8942fd-34dd-4b5b-ba05-0c6488374820","Type":"ContainerStarted","Data":"db27efb6233d3e713cd58c2a884e9ee3424c06428b10764b5dacd35d63501086"}
Jul 17 12:30:09.262024 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:09.261936    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-99.ec2.internal" podStartSLOduration=17.261919695 podStartE2EDuration="17.261919695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:30:09.261785987 +0000 UTC m=+18.841898029" watchObservedRunningTime="2026-07-17 12:30:09.261919695 +0000 UTC m=+18.842031714"
Jul 17 12:30:09.415734 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:09.415672    2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 17 12:30:09.545341 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:09.545218    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:30:09.906255 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:09.905340    2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T12:30:09.415714361Z","UUID":"985f2da3-03d9-47bb-b0ba-9f7bd1381880","Handler":null,"Name":"","Endpoint":""}
Jul 17 12:30:09.907915 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:09.907459    2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 17 12:30:09.907915 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:09.907503    2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 17 12:30:10.009391 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:10.009361    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:30:10.009585 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:10.009495    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xl2np" podUID="a72fcab3-1981-4637-a041-225335390f1d"
Jul 17 12:30:10.009849 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:10.009361    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:30:10.009849 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:10.009812    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b4b4" podUID="c803ba62-1a8b-4b71-a355-fc4427be817b"
Jul 17 12:30:10.250964 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:10.250876    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsvdx" event={"ID":"83656ae5-efe4-426a-9a4c-080b18541deb","Type":"ContainerStarted","Data":"a5a0cf6eea10c2d2a8cb08bae48dc1d9e75247b023c072722688f47986b2c6b2"}
Jul 17 12:30:10.880783 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:10.880738    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5v7sc"
Jul 17 12:30:10.881607 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:10.881582    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5v7sc"
Jul 17 12:30:10.908142 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:10.908076    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xdqkr" podStartSLOduration=6.388477363 podStartE2EDuration="19.908053216s" totalImagesPullingTime="13.519575853s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:29:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:30:09.290174363 +0000 UTC m=+18.870286410" watchObservedRunningTime="2026-07-17 12:30:10.908053216 +0000 UTC m=+20.488165237"
Jul 17 12:30:11.009829 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:11.009793    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:30:11.009994 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:11.009948    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxrxw" podUID="937e7356-528f-4a56-a80e-9486657ec301"
Jul 17 12:30:11.255863 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:11.255771    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsvdx" event={"ID":"83656ae5-efe4-426a-9a4c-080b18541deb","Type":"ContainerStarted","Data":"96834e93f297042ac2abba05ae6f475d4dc5d65e751e50a023aa005b7916227a"}
Jul 17 12:30:11.255863 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:11.255846    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5v7sc"
Jul 17 12:30:11.256545 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:11.256524    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5v7sc"
Jul 17 12:30:11.300070 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:11.300009    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsvdx" podStartSLOduration=4.28546952 podStartE2EDuration="20.299988663s" totalImagesPullingTime="16.014519143s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:29:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:30:11.299744291 +0000 UTC m=+20.879856335" watchObservedRunningTime="2026-07-17 12:30:11.299988663 +0000 UTC m=+20.880100683"
Jul 17 12:30:12.009445 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:12.009407    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:30:12.009642 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:12.009459    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:30:12.009642 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:12.009570    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b4b4" podUID="c803ba62-1a8b-4b71-a355-fc4427be817b"
Jul 17 12:30:12.009813 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:12.009724    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xl2np" podUID="a72fcab3-1981-4637-a041-225335390f1d"
Jul 17 12:30:13.008980 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:13.008942    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:30:13.009462 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:13.009104    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxrxw" podUID="937e7356-528f-4a56-a80e-9486657ec301"
Jul 17 12:30:13.262592 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:13.262470    2564 generic.go:356] "Generic (PLEG): container finished" podID="b09e01d5-4dca-4a57-903d-f6daacfdb5d9" containerID="9e57d29b50f287d73b4f1b22e5475d0d04e8c09702240f1d8389c12ec6607152" exitCode=0
Jul 17 12:30:13.262592 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:13.262562    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sld25" event={"ID":"b09e01d5-4dca-4a57-903d-f6daacfdb5d9","Type":"ContainerDied","Data":"9e57d29b50f287d73b4f1b22e5475d0d04e8c09702240f1d8389c12ec6607152"}
Jul 17 12:30:14.009136 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:14.009099    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:30:14.009766 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:14.009099    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:30:14.009766 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:14.009248    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xl2np" podUID="a72fcab3-1981-4637-a041-225335390f1d"
Jul 17 12:30:14.009766 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:14.009321    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b4b4" podUID="c803ba62-1a8b-4b71-a355-fc4427be817b"
Jul 17 12:30:15.009223 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:15.009184    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:30:15.009640 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:15.009341    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxrxw" podUID="937e7356-528f-4a56-a80e-9486657ec301"
Jul 17 12:30:16.008648 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:16.008613    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:30:16.008844 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:16.008615    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:30:16.008844 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:16.008768    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b4b4" podUID="c803ba62-1a8b-4b71-a355-fc4427be817b"
Jul 17 12:30:16.008844 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:16.008815    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xl2np" podUID="a72fcab3-1981-4637-a041-225335390f1d"
Jul 17 12:30:17.009370 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:17.009335    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:30:17.009851 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:17.009485    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxrxw" podUID="937e7356-528f-4a56-a80e-9486657ec301"
Jul 17 12:30:18.009336 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:18.009301    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:30:18.009540 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:18.009303    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:30:18.009540 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:18.009428    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xl2np" podUID="a72fcab3-1981-4637-a041-225335390f1d"
Jul 17 12:30:18.010007 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:18.009528    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b4b4" podUID="c803ba62-1a8b-4b71-a355-fc4427be817b"
Jul 17 12:30:19.009474 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:19.009450    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:30:19.009596 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:19.009569    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxrxw" podUID="937e7356-528f-4a56-a80e-9486657ec301"
Jul 17 12:30:19.278484 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:19.278315    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f958_7ab622df-86d3-4cc3-aba7-123f4bc0c996/ovn-acl-logging/0.log"
Jul 17 12:30:19.278816 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:19.278790    2564 generic.go:356] "Generic (PLEG): container finished" podID="7ab622df-86d3-4cc3-aba7-123f4bc0c996" containerID="565efe98a78e49a3a409630aa0f45d6c8446c01f6e5a75a1e937cd11ae19310c" exitCode=1
Jul 17 12:30:19.278933 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:19.278864    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9f958" event={"ID":"7ab622df-86d3-4cc3-aba7-123f4bc0c996","Type":"ContainerStarted","Data":"8d990b0aa57dbc18cd7cf9dccdfb3ec1a6a65bac697212395a9d9eb1852aab53"}
Jul 17 12:30:19.278933 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:19.278930    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9f958" event={"ID":"7ab622df-86d3-4cc3-aba7-123f4bc0c996","Type":"ContainerStarted","Data":"0e31c6fcde05e9cf4818693724f140990a24a0a097e2451adde8707488770ed8"}
Jul 17 12:30:19.279018 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:19.278962    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9f958" event={"ID":"7ab622df-86d3-4cc3-aba7-123f4bc0c996","Type":"ContainerStarted","Data":"b3c5664ad5e77dd9a4f57b2df4ce3bfb6a59b86afdf7e5da97aa226c0714dd44"}
Jul 17 12:30:19.279018 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:19.279006    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9f958" event={"ID":"7ab622df-86d3-4cc3-aba7-123f4bc0c996","Type":"ContainerStarted","Data":"3be5ef880012f450c85d4d3082442c7e41b7f15cb5ddea617b6443356ca742ed"}
Jul 17 12:30:19.279091 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:19.279038    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9f958" event={"ID":"7ab622df-86d3-4cc3-aba7-123f4bc0c996","Type":"ContainerDied","Data":"565efe98a78e49a3a409630aa0f45d6c8446c01f6e5a75a1e937cd11ae19310c"}
Jul 17 12:30:19.279091 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:19.279082    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9f958" event={"ID":"7ab622df-86d3-4cc3-aba7-123f4bc0c996","Type":"ContainerStarted","Data":"88b3d23055cc85c21f4996580f9115dbefa64badffc84d6c30856ce1221c9cd4"}
Jul 17 12:30:19.280989 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:19.280964    2564 generic.go:356] "Generic (PLEG): container finished" podID="b09e01d5-4dca-4a57-903d-f6daacfdb5d9" containerID="4b565ab1a7e46ac5da2ec2cd4011e78929dfcb63bbb077aa71290f46225ebb23" exitCode=0
Jul 17 12:30:19.281111 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:19.281005    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sld25" event={"ID":"b09e01d5-4dca-4a57-903d-f6daacfdb5d9","Type":"ContainerDied","Data":"4b565ab1a7e46ac5da2ec2cd4011e78929dfcb63bbb077aa71290f46225ebb23"}
Jul 17 12:30:20.009151 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:20.009116    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:30:20.009294 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:20.009116    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:30:20.009294 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:20.009241    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xl2np" podUID="a72fcab3-1981-4637-a041-225335390f1d"
Jul 17 12:30:20.009294 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:20.009289    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b4b4" podUID="c803ba62-1a8b-4b71-a355-fc4427be817b"
Jul 17 12:30:21.009971 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:21.009939    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:30:21.010445 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:21.010045    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxrxw" podUID="937e7356-528f-4a56-a80e-9486657ec301"
Jul 17 12:30:21.288626 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:21.288539    2564 generic.go:356] "Generic (PLEG): container finished" podID="b09e01d5-4dca-4a57-903d-f6daacfdb5d9" containerID="a76251281a4e378a1d8b737bbd6252ab16369f9962d1e85e1ba104821e40a017" exitCode=0
Jul 17 12:30:21.288626 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:21.288590    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sld25" event={"ID":"b09e01d5-4dca-4a57-903d-f6daacfdb5d9","Type":"ContainerDied","Data":"a76251281a4e378a1d8b737bbd6252ab16369f9962d1e85e1ba104821e40a017"}
Jul 17 12:30:22.009187 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:22.009153    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:30:22.009384 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:22.009280    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xl2np" podUID="a72fcab3-1981-4637-a041-225335390f1d"
Jul 17 12:30:22.009384 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:22.009326    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:30:22.009498 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:22.009414    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b4b4" podUID="c803ba62-1a8b-4b71-a355-fc4427be817b"
Jul 17 12:30:22.294273 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:22.294202    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f958_7ab622df-86d3-4cc3-aba7-123f4bc0c996/ovn-acl-logging/0.log"
Jul 17 12:30:22.294857 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:22.294598    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9f958" event={"ID":"7ab622df-86d3-4cc3-aba7-123f4bc0c996","Type":"ContainerStarted","Data":"964074ed2692e5fceeb52515d4d18570b7f6d0e9ca348151067570a48727226a"}
Jul 17 12:30:23.008770 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:23.008732    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:30:23.008942 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:23.008856    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxrxw" podUID="937e7356-528f-4a56-a80e-9486657ec301"
Jul 17 12:30:24.008982 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:24.008781    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:30:24.009455 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:24.008778    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:30:24.009455 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:24.009070    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xl2np" podUID="a72fcab3-1981-4637-a041-225335390f1d"
Jul 17 12:30:24.009455 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:24.009137    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b4b4" podUID="c803ba62-1a8b-4b71-a355-fc4427be817b"
Jul 17 12:30:24.303438 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:24.303360    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f958_7ab622df-86d3-4cc3-aba7-123f4bc0c996/ovn-acl-logging/0.log"
Jul 17 12:30:24.303812 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:24.303780    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9f958" event={"ID":"7ab622df-86d3-4cc3-aba7-123f4bc0c996","Type":"ContainerStarted","Data":"df2f4d6311e8c82d785550c90b92b47070f3caf595f25ab95ce315911dc7af5e"}
Jul 17 12:30:24.304082 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:24.304065    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:30:24.304233 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:24.304208    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:30:24.304325 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:24.304280    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:30:24.304374 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:24.304329    2564 scope.go:117] "RemoveContainer" containerID="565efe98a78e49a3a409630aa0f45d6c8446c01f6e5a75a1e937cd11ae19310c"
Jul 17 12:30:24.321493 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:24.321465    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:30:24.326240 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:24.326216    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:30:24.651317 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:24.651232    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c803ba62-1a8b-4b71-a355-fc4427be817b-original-pull-secret\") pod \"global-pull-secret-syncer-9b4b4\" (UID: \"c803ba62-1a8b-4b71-a355-fc4427be817b\") " pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:30:24.651317 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:24.651300    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/937e7356-528f-4a56-a80e-9486657ec301-metrics-certs\") pod \"network-metrics-daemon-lxrxw\" (UID: \"937e7356-528f-4a56-a80e-9486657ec301\") " pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:30:24.651540 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:24.651397    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 12:30:24.651540 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:24.651461    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c803ba62-1a8b-4b71-a355-fc4427be817b-original-pull-secret podName:c803ba62-1a8b-4b71-a355-fc4427be817b nodeName:}" failed. No retries permitted until 2026-07-17 12:30:56.651445344 +0000 UTC m=+66.231557346 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c803ba62-1a8b-4b71-a355-fc4427be817b-original-pull-secret") pod "global-pull-secret-syncer-9b4b4" (UID: "c803ba62-1a8b-4b71-a355-fc4427be817b") : object "kube-system"/"original-pull-secret" not registered
Jul 17 12:30:24.651540 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:24.651397    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:30:24.651540 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:24.651504    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/937e7356-528f-4a56-a80e-9486657ec301-metrics-certs podName:937e7356-528f-4a56-a80e-9486657ec301 nodeName:}" failed. No retries permitted until 2026-07-17 12:30:56.651493641 +0000 UTC m=+66.231605646 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/937e7356-528f-4a56-a80e-9486657ec301-metrics-certs") pod "network-metrics-daemon-lxrxw" (UID: "937e7356-528f-4a56-a80e-9486657ec301") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:30:24.752395 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:24.752359    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmwjl\" (UniqueName: \"kubernetes.io/projected/a72fcab3-1981-4637-a041-225335390f1d-kube-api-access-pmwjl\") pod \"network-check-target-xl2np\" (UID: \"a72fcab3-1981-4637-a041-225335390f1d\") " pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:30:24.752597 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:24.752535    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 12:30:24.752597 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:24.752556    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 12:30:24.752597 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:24.752568    2564 projected.go:196] Error preparing data for projected volume kube-api-access-pmwjl for pod openshift-network-diagnostics/network-check-target-xl2np: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:30:24.752773 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:24.752634    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a72fcab3-1981-4637-a041-225335390f1d-kube-api-access-pmwjl podName:a72fcab3-1981-4637-a041-225335390f1d nodeName:}" failed. No retries permitted until 2026-07-17 12:30:56.752614215 +0000 UTC m=+66.332726232 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pmwjl" (UniqueName: "kubernetes.io/projected/a72fcab3-1981-4637-a041-225335390f1d-kube-api-access-pmwjl") pod "network-check-target-xl2np" (UID: "a72fcab3-1981-4637-a041-225335390f1d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:30:25.009483 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:25.009395    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:30:25.009938 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:25.009527    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxrxw" podUID="937e7356-528f-4a56-a80e-9486657ec301"
Jul 17 12:30:25.586572 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:25.586533    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lxrxw"]
Jul 17 12:30:25.586852 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:25.586692    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:30:25.586852 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:25.586830    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxrxw" podUID="937e7356-528f-4a56-a80e-9486657ec301"
Jul 17 12:30:25.589719 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:25.589676    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9b4b4"]
Jul 17 12:30:25.589850 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:25.589810    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:30:25.589936 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:25.589914    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b4b4" podUID="c803ba62-1a8b-4b71-a355-fc4427be817b"
Jul 17 12:30:25.590447 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:25.590417    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xl2np"]
Jul 17 12:30:25.590568 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:25.590501    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:30:25.590632 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:25.590590    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xl2np" podUID="a72fcab3-1981-4637-a041-225335390f1d"
Jul 17 12:30:26.313418 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:26.313253    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f958_7ab622df-86d3-4cc3-aba7-123f4bc0c996/ovn-acl-logging/0.log"
Jul 17 12:30:26.313903 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:26.313877    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9f958" event={"ID":"7ab622df-86d3-4cc3-aba7-123f4bc0c996","Type":"ContainerStarted","Data":"bf0cbda2abaf3bc0446ca5109ad25a9759c0041347a2dd15042b1efc1daa76b9"}
Jul 17 12:30:26.316587 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:26.316554    2564 generic.go:356] "Generic (PLEG): container finished" podID="b09e01d5-4dca-4a57-903d-f6daacfdb5d9" containerID="068c652d654c65eee7ce63f5b15fa45f897f857c83ae98290ee13160a3be6143" exitCode=0
Jul 17 12:30:26.316587 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:26.316582    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sld25" event={"ID":"b09e01d5-4dca-4a57-903d-f6daacfdb5d9","Type":"ContainerDied","Data":"068c652d654c65eee7ce63f5b15fa45f897f857c83ae98290ee13160a3be6143"}
Jul 17 12:30:26.348513 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:26.348462    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9f958" podStartSLOduration=8.939638566 podStartE2EDuration="34.348444259s" totalImagesPullingTime="25.408805693s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:29:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=2 observedRunningTime="2026-07-17 12:30:26.347757335 +0000 UTC m=+35.927869381" watchObservedRunningTime="2026-07-17 12:30:26.348444259 +0000 UTC m=+35.928556700"
Jul 17 12:30:27.008789 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:27.008749    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:30:27.008789 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:27.008775    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:30:27.009061 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:27.008750    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:30:27.009061 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:27.008893    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxrxw" podUID="937e7356-528f-4a56-a80e-9486657ec301"
Jul 17 12:30:27.009061 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:27.008991    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xl2np" podUID="a72fcab3-1981-4637-a041-225335390f1d"
Jul 17 12:30:27.009213 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:27.009067    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b4b4" podUID="c803ba62-1a8b-4b71-a355-fc4427be817b"
Jul 17 12:30:27.322267 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:27.322170    2564 generic.go:356] "Generic (PLEG): container finished" podID="b09e01d5-4dca-4a57-903d-f6daacfdb5d9" containerID="0e49cc6c6e3b41319679bdb48b535a29e87e729f0bd80a6c096633b7d38aa9fc" exitCode=0
Jul 17 12:30:27.322267 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:27.322245    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sld25" event={"ID":"b09e01d5-4dca-4a57-903d-f6daacfdb5d9","Type":"ContainerDied","Data":"0e49cc6c6e3b41319679bdb48b535a29e87e729f0bd80a6c096633b7d38aa9fc"}
Jul 17 12:30:28.328454 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:28.328414    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sld25" event={"ID":"b09e01d5-4dca-4a57-903d-f6daacfdb5d9","Type":"ContainerStarted","Data":"59a215b775b98196791ddf02b5a6e68066fe1bd77589c63efc9d90c5cce9b62b"}
Jul 17 12:30:28.354482 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:28.354429    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sld25" podStartSLOduration=7.450483849 podStartE2EDuration="36.354413847s" totalImagesPullingTime="28.903929998s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:29:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:30:28.353971644 +0000 UTC m=+37.934083664" watchObservedRunningTime="2026-07-17 12:30:28.354413847 +0000 UTC m=+37.934525866"
Jul 17 12:30:29.008611 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:29.008572    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:30:29.008776 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:29.008589    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:30:29.008776 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:29.008589    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:30:29.008776 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:29.008685    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxrxw" podUID="937e7356-528f-4a56-a80e-9486657ec301"
Jul 17 12:30:29.008776 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:29.008758    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xl2np" podUID="a72fcab3-1981-4637-a041-225335390f1d"
Jul 17 12:30:29.008982 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:29.008850    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b4b4" podUID="c803ba62-1a8b-4b71-a355-fc4427be817b"
Jul 17 12:30:31.009922 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:31.009888    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:30:31.010508 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:31.009958    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:30:31.010508 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:31.009973    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xl2np" podUID="a72fcab3-1981-4637-a041-225335390f1d"
Jul 17 12:30:31.010508 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:31.010082    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lxrxw" podUID="937e7356-528f-4a56-a80e-9486657ec301"
Jul 17 12:30:31.010508 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:31.010112    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:30:31.010508 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:31.010171    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b4b4" podUID="c803ba62-1a8b-4b71-a355-fc4427be817b"
Jul 17 12:30:31.222381 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:31.222299    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-99.ec2.internal" event="NodeReady"
Jul 17 12:30:31.222526 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:31.222400    2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 17 12:30:31.320179 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:31.320141    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4h4tb"]
Jul 17 12:30:31.324076 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:31.324054    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wkj54"]
Jul 17 12:30:31.324211 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:31.324189    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4h4tb"
Jul 17 12:30:31.327507 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:31.327478    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"ingress-canary-4h4tb\" is forbidden: User \"system:node:ip-10-0-137-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-137-99.ec2.internal' and this object" podUID="ed27f8cc-6874-4594-905b-2483d9f9463d" pod="openshift-ingress-canary/ingress-canary-4h4tb"
Jul 17 12:30:31.327507 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:31.327488    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-dxp65\" is forbidden: User \"system:node:ip-10-0-137-99.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-137-99.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dxp65\"" type="*v1.Secret"
Jul 17 12:30:31.327874 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:31.327852    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"canary-serving-cert\" is forbidden: User \"system:node:ip-10-0-137-99.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-137-99.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret"
Jul 17 12:30:31.327967 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:31.327951    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-137-99.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-137-99.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Jul 17 12:30:31.328194 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:31.328173    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wkj54"
Jul 17 12:30:31.328462 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:31.328431    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-137-99.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-137-99.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Jul 17 12:30:31.330509 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:31.330483    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"ingress-canary-4h4tb\" is forbidden: User \"system:node:ip-10-0-137-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-137-99.ec2.internal' and this object" podUID="ed27f8cc-6874-4594-905b-2483d9f9463d" pod="openshift-ingress-canary/ingress-canary-4h4tb"
Jul 17 12:30:31.331103 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:31.331086    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 17 12:30:31.331487 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:31.331474    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bjllw\""
Jul 17 12:30:31.334862 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:31.334845    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 17 12:30:31.350799 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:31.350770    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4h4tb"]
Jul 17 12:30:31.366892 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:31.366863    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wkj54"]
Jul 17 12:30:31.498571 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:31.498471    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/efb8a146-b5cf-4b54-bb64-3b060a736ca4-metrics-tls\") pod \"dns-default-wkj54\" (UID: \"efb8a146-b5cf-4b54-bb64-3b060a736ca4\") " pod="openshift-dns/dns-default-wkj54"
Jul 17 12:30:31.498571 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:31.498534    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtlsf\" (UniqueName: \"kubernetes.io/projected/ed27f8cc-6874-4594-905b-2483d9f9463d-kube-api-access-dtlsf\") pod \"ingress-canary-4h4tb\" (UID: \"ed27f8cc-6874-4594-905b-2483d9f9463d\") " pod="openshift-ingress-canary/ingress-canary-4h4tb"
Jul 17 12:30:31.498836 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:31.498614    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efb8a146-b5cf-4b54-bb64-3b060a736ca4-tmp-dir\") pod \"dns-default-wkj54\" (UID: \"efb8a146-b5cf-4b54-bb64-3b060a736ca4\") " pod="openshift-dns/dns-default-wkj54"
Jul 17 12:30:31.498836 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:31.498667    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/efb8a146-b5cf-4b54-bb64-3b060a736ca4-config-volume\") pod \"dns-default-wkj54\" (UID: \"efb8a146-b5cf-4b54-bb64-3b060a736ca4\") " pod="openshift-dns/dns-default-wkj54"
Jul 17 12:30:31.498836 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:31.498732    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c48gm\" (UniqueName: \"kubernetes.io/projected/efb8a146-b5cf-4b54-bb64-3b060a736ca4-kube-api-access-c48gm\") pod \"dns-default-wkj54\" (UID: \"efb8a146-b5cf-4b54-bb64-3b060a736ca4\") " pod="openshift-dns/dns-default-wkj54"
Jul 17 12:30:31.498836 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:31.498796    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed27f8cc-6874-4594-905b-2483d9f9463d-cert\") pod \"ingress-canary-4h4tb\" (UID: \"ed27f8cc-6874-4594-905b-2483d9f9463d\") " pod="openshift-ingress-canary/ingress-canary-4h4tb"
Jul 17 12:30:31.599614 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:31.599580    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed27f8cc-6874-4594-905b-2483d9f9463d-cert\") pod \"ingress-canary-4h4tb\" (UID: \"ed27f8cc-6874-4594-905b-2483d9f9463d\") " pod="openshift-ingress-canary/ingress-canary-4h4tb"
Jul 17 12:30:31.599614 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:31.599618    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/efb8a146-b5cf-4b54-bb64-3b060a736ca4-metrics-tls\") pod \"dns-default-wkj54\" (UID: \"efb8a146-b5cf-4b54-bb64-3b060a736ca4\") " pod="openshift-dns/dns-default-wkj54"
Jul 17 12:30:31.599858 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:31.599648    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtlsf\" (UniqueName: \"kubernetes.io/projected/ed27f8cc-6874-4594-905b-2483d9f9463d-kube-api-access-dtlsf\") pod \"ingress-canary-4h4tb\" (UID: \"ed27f8cc-6874-4594-905b-2483d9f9463d\") " pod="openshift-ingress-canary/ingress-canary-4h4tb"
Jul 17 12:30:31.599858 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:31.599689    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efb8a146-b5cf-4b54-bb64-3b060a736ca4-tmp-dir\") pod \"dns-default-wkj54\" (UID: \"efb8a146-b5cf-4b54-bb64-3b060a736ca4\") " pod="openshift-dns/dns-default-wkj54"
Jul 17 12:30:31.599858 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:31.599740    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/efb8a146-b5cf-4b54-bb64-3b060a736ca4-config-volume\") pod \"dns-default-wkj54\" (UID: \"efb8a146-b5cf-4b54-bb64-3b060a736ca4\") " pod="openshift-dns/dns-default-wkj54"
Jul 17 12:30:31.599858 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:31.599757    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 12:30:31.599858 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:31.599775    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c48gm\" (UniqueName: \"kubernetes.io/projected/efb8a146-b5cf-4b54-bb64-3b060a736ca4-kube-api-access-c48gm\") pod \"dns-default-wkj54\" (UID: \"efb8a146-b5cf-4b54-bb64-3b060a736ca4\") " pod="openshift-dns/dns-default-wkj54"
Jul 17 12:30:31.599858 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:31.599831    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efb8a146-b5cf-4b54-bb64-3b060a736ca4-metrics-tls podName:efb8a146-b5cf-4b54-bb64-3b060a736ca4 nodeName:}" failed. No retries permitted until 2026-07-17 12:30:32.099813238 +0000 UTC m=+41.679925240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/efb8a146-b5cf-4b54-bb64-3b060a736ca4-metrics-tls") pod "dns-default-wkj54" (UID: "efb8a146-b5cf-4b54-bb64-3b060a736ca4") : secret "dns-default-metrics-tls" not found
Jul 17 12:30:31.600202 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:31.600168    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efb8a146-b5cf-4b54-bb64-3b060a736ca4-tmp-dir\") pod \"dns-default-wkj54\" (UID: \"efb8a146-b5cf-4b54-bb64-3b060a736ca4\") " pod="openshift-dns/dns-default-wkj54"
Jul 17 12:30:31.600400 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:31.600380    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/efb8a146-b5cf-4b54-bb64-3b060a736ca4-config-volume\") pod \"dns-default-wkj54\" (UID: \"efb8a146-b5cf-4b54-bb64-3b060a736ca4\") " pod="openshift-dns/dns-default-wkj54"
Jul 17 12:30:31.611153 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:31.611127    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c48gm\" (UniqueName: \"kubernetes.io/projected/efb8a146-b5cf-4b54-bb64-3b060a736ca4-kube-api-access-c48gm\") pod \"dns-default-wkj54\" (UID: \"efb8a146-b5cf-4b54-bb64-3b060a736ca4\") " pod="openshift-dns/dns-default-wkj54"
Jul 17 12:30:32.104642 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:32.104602    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/efb8a146-b5cf-4b54-bb64-3b060a736ca4-metrics-tls\") pod \"dns-default-wkj54\" (UID: \"efb8a146-b5cf-4b54-bb64-3b060a736ca4\") " pod="openshift-dns/dns-default-wkj54"
Jul 17 12:30:32.105076 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:32.104776    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 12:30:32.105076 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:32.104858    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efb8a146-b5cf-4b54-bb64-3b060a736ca4-metrics-tls podName:efb8a146-b5cf-4b54-bb64-3b060a736ca4 nodeName:}" failed. No retries permitted until 2026-07-17 12:30:33.104842258 +0000 UTC m=+42.684954255 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/efb8a146-b5cf-4b54-bb64-3b060a736ca4-metrics-tls") pod "dns-default-wkj54" (UID: "efb8a146-b5cf-4b54-bb64-3b060a736ca4") : secret "dns-default-metrics-tls" not found
Jul 17 12:30:32.180489 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:32.180459    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 17 12:30:32.247085 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:32.247054    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 17 12:30:32.255135 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:32.255091    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtlsf\" (UniqueName: \"kubernetes.io/projected/ed27f8cc-6874-4594-905b-2483d9f9463d-kube-api-access-dtlsf\") pod \"ingress-canary-4h4tb\" (UID: \"ed27f8cc-6874-4594-905b-2483d9f9463d\") " pod="openshift-ingress-canary/ingress-canary-4h4tb"
Jul 17 12:30:32.460610 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:32.460524    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 17 12:30:32.472859 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:32.472828    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed27f8cc-6874-4594-905b-2483d9f9463d-cert\") pod \"ingress-canary-4h4tb\" (UID: \"ed27f8cc-6874-4594-905b-2483d9f9463d\") " pod="openshift-ingress-canary/ingress-canary-4h4tb"
Jul 17 12:30:32.807469 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:32.807376    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dxp65\""
Jul 17 12:30:32.815903 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:32.815879    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4h4tb"
Jul 17 12:30:33.008842 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:33.008645    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:30:33.008990 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:33.008646    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:30:33.009040 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:33.008689    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:30:33.013314 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:33.013291    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 12:30:33.013537 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:33.013523    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 12:30:33.013708 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:33.013683    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 12:30:33.013857 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:33.013840    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-x9krt\""
Jul 17 12:30:33.014827 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:33.014806    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 12:30:33.014927 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:33.014832    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wrsj9\""
Jul 17 12:30:33.063592 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:33.063460    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4h4tb"]
Jul 17 12:30:33.067569 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:30:33.067536    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded27f8cc_6874_4594_905b_2483d9f9463d.slice/crio-3535a95b15b3ff02273b944eca72f9f1dda451add9b142e303949da4cae3c112 WatchSource:0}: Error finding container 3535a95b15b3ff02273b944eca72f9f1dda451add9b142e303949da4cae3c112: Status 404 returned error can't find the container with id 3535a95b15b3ff02273b944eca72f9f1dda451add9b142e303949da4cae3c112
Jul 17 12:30:33.113542 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:33.113498    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/efb8a146-b5cf-4b54-bb64-3b060a736ca4-metrics-tls\") pod \"dns-default-wkj54\" (UID: \"efb8a146-b5cf-4b54-bb64-3b060a736ca4\") " pod="openshift-dns/dns-default-wkj54"
Jul 17 12:30:33.114058 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:33.113636    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 12:30:33.114058 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:33.113733    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efb8a146-b5cf-4b54-bb64-3b060a736ca4-metrics-tls podName:efb8a146-b5cf-4b54-bb64-3b060a736ca4 nodeName:}" failed. No retries permitted until 2026-07-17 12:30:35.113690686 +0000 UTC m=+44.693802701 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/efb8a146-b5cf-4b54-bb64-3b060a736ca4-metrics-tls") pod "dns-default-wkj54" (UID: "efb8a146-b5cf-4b54-bb64-3b060a736ca4") : secret "dns-default-metrics-tls" not found
Jul 17 12:30:33.341617 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:33.341535    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4h4tb" event={"ID":"ed27f8cc-6874-4594-905b-2483d9f9463d","Type":"ContainerStarted","Data":"3535a95b15b3ff02273b944eca72f9f1dda451add9b142e303949da4cae3c112"}
Jul 17 12:30:35.129887 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:35.129848    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/efb8a146-b5cf-4b54-bb64-3b060a736ca4-metrics-tls\") pod \"dns-default-wkj54\" (UID: \"efb8a146-b5cf-4b54-bb64-3b060a736ca4\") " pod="openshift-dns/dns-default-wkj54"
Jul 17 12:30:35.130259 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:35.130023    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 12:30:35.130259 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:35.130091    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efb8a146-b5cf-4b54-bb64-3b060a736ca4-metrics-tls podName:efb8a146-b5cf-4b54-bb64-3b060a736ca4 nodeName:}" failed. No retries permitted until 2026-07-17 12:30:39.130075718 +0000 UTC m=+48.710187715 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/efb8a146-b5cf-4b54-bb64-3b060a736ca4-metrics-tls") pod "dns-default-wkj54" (UID: "efb8a146-b5cf-4b54-bb64-3b060a736ca4") : secret "dns-default-metrics-tls" not found
Jul 17 12:30:36.350980 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:36.350942    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4h4tb" event={"ID":"ed27f8cc-6874-4594-905b-2483d9f9463d","Type":"ContainerStarted","Data":"e3f85411434f6785682483ca24864c14353c4bcc3e7a6a6b2330a27cf5862513"}
Jul 17 12:30:36.369913 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:36.369862    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4h4tb" podStartSLOduration=2.4713908 podStartE2EDuration="5.369846619s" totalImagesPullingTime="2.898455819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:30:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:30:36.369362872 +0000 UTC m=+45.949474892" watchObservedRunningTime="2026-07-17 12:30:36.369846619 +0000 UTC m=+45.949958672"
Jul 17 12:30:39.155692 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:39.155647    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/efb8a146-b5cf-4b54-bb64-3b060a736ca4-metrics-tls\") pod \"dns-default-wkj54\" (UID: \"efb8a146-b5cf-4b54-bb64-3b060a736ca4\") " pod="openshift-dns/dns-default-wkj54"
Jul 17 12:30:39.156089 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:39.155842    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 12:30:39.156089 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:39.155907    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efb8a146-b5cf-4b54-bb64-3b060a736ca4-metrics-tls podName:efb8a146-b5cf-4b54-bb64-3b060a736ca4 nodeName:}" failed. No retries permitted until 2026-07-17 12:30:47.155893603 +0000 UTC m=+56.736005605 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/efb8a146-b5cf-4b54-bb64-3b060a736ca4-metrics-tls") pod "dns-default-wkj54" (UID: "efb8a146-b5cf-4b54-bb64-3b060a736ca4") : secret "dns-default-metrics-tls" not found
Jul 17 12:30:47.209424 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:47.209368    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/efb8a146-b5cf-4b54-bb64-3b060a736ca4-metrics-tls\") pod \"dns-default-wkj54\" (UID: \"efb8a146-b5cf-4b54-bb64-3b060a736ca4\") " pod="openshift-dns/dns-default-wkj54"
Jul 17 12:30:47.210016 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:47.209551    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 12:30:47.210016 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:47.209644    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efb8a146-b5cf-4b54-bb64-3b060a736ca4-metrics-tls podName:efb8a146-b5cf-4b54-bb64-3b060a736ca4 nodeName:}" failed. No retries permitted until 2026-07-17 12:31:03.209624805 +0000 UTC m=+72.789736802 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/efb8a146-b5cf-4b54-bb64-3b060a736ca4-metrics-tls") pod "dns-default-wkj54" (UID: "efb8a146-b5cf-4b54-bb64-3b060a736ca4") : secret "dns-default-metrics-tls" not found
Jul 17 12:30:56.668887 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:56.668845    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c803ba62-1a8b-4b71-a355-fc4427be817b-original-pull-secret\") pod \"global-pull-secret-syncer-9b4b4\" (UID: \"c803ba62-1a8b-4b71-a355-fc4427be817b\") " pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:30:56.669269 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:56.668908    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/937e7356-528f-4a56-a80e-9486657ec301-metrics-certs\") pod \"network-metrics-daemon-lxrxw\" (UID: \"937e7356-528f-4a56-a80e-9486657ec301\") " pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:30:56.671908 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:56.671886    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 12:30:56.672023 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:56.672006    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 12:30:56.679230 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:56.679206    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 17 12:30:56.679294 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:30:56.679266    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/937e7356-528f-4a56-a80e-9486657ec301-metrics-certs podName:937e7356-528f-4a56-a80e-9486657ec301 nodeName:}" failed. No retries permitted until 2026-07-17 12:32:00.679251879 +0000 UTC m=+130.259363875 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/937e7356-528f-4a56-a80e-9486657ec301-metrics-certs") pod "network-metrics-daemon-lxrxw" (UID: "937e7356-528f-4a56-a80e-9486657ec301") : secret "metrics-daemon-secret" not found
Jul 17 12:30:56.683254 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:56.683232    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c803ba62-1a8b-4b71-a355-fc4427be817b-original-pull-secret\") pod \"global-pull-secret-syncer-9b4b4\" (UID: \"c803ba62-1a8b-4b71-a355-fc4427be817b\") " pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:30:56.726160 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:56.726123    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b4b4"
Jul 17 12:30:56.769944 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:56.769908    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmwjl\" (UniqueName: \"kubernetes.io/projected/a72fcab3-1981-4637-a041-225335390f1d-kube-api-access-pmwjl\") pod \"network-check-target-xl2np\" (UID: \"a72fcab3-1981-4637-a041-225335390f1d\") " pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:30:56.773294 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:56.773271    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 12:30:56.783772 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:56.783743    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 12:30:56.794323 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:56.794293    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmwjl\" (UniqueName: \"kubernetes.io/projected/a72fcab3-1981-4637-a041-225335390f1d-kube-api-access-pmwjl\") pod \"network-check-target-xl2np\" (UID: \"a72fcab3-1981-4637-a041-225335390f1d\") " pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:30:56.894422 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:56.894388    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9b4b4"]
Jul 17 12:30:56.897663 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:30:56.897631    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc803ba62_1a8b_4b71_a355_fc4427be817b.slice/crio-362cebe5f5a040d05d11c4a48128ae2d517890d68504e76f43d1fa22b0f84826 WatchSource:0}: Error finding container 362cebe5f5a040d05d11c4a48128ae2d517890d68504e76f43d1fa22b0f84826: Status 404 returned error can't find the container with id 362cebe5f5a040d05d11c4a48128ae2d517890d68504e76f43d1fa22b0f84826
Jul 17 12:30:57.033330 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:57.033301    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-x9krt\""
Jul 17 12:30:57.041318 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:57.041296    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:30:57.191854 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:57.191820    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xl2np"]
Jul 17 12:30:57.195721 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:30:57.195682    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda72fcab3_1981_4637_a041_225335390f1d.slice/crio-4716d93603895a52ef187eef98ade6284aa1232a00dfd7df10ca241369d0a31c WatchSource:0}: Error finding container 4716d93603895a52ef187eef98ade6284aa1232a00dfd7df10ca241369d0a31c: Status 404 returned error can't find the container with id 4716d93603895a52ef187eef98ade6284aa1232a00dfd7df10ca241369d0a31c
Jul 17 12:30:57.344308 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:57.344230    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9f958"
Jul 17 12:30:57.402218 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:57.402168    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9b4b4" event={"ID":"c803ba62-1a8b-4b71-a355-fc4427be817b","Type":"ContainerStarted","Data":"362cebe5f5a040d05d11c4a48128ae2d517890d68504e76f43d1fa22b0f84826"}
Jul 17 12:30:57.403342 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:30:57.403311    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xl2np" event={"ID":"a72fcab3-1981-4637-a041-225335390f1d","Type":"ContainerStarted","Data":"4716d93603895a52ef187eef98ade6284aa1232a00dfd7df10ca241369d0a31c"}
Jul 17 12:31:02.418179 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:02.418141    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xl2np" event={"ID":"a72fcab3-1981-4637-a041-225335390f1d","Type":"ContainerStarted","Data":"3baf286365eabfd63e82e5b727b47165267bf760fcefccaaeec664b1475e26cb"}
Jul 17 12:31:02.418654 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:02.418267    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:31:02.419654 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:02.419630    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9b4b4" event={"ID":"c803ba62-1a8b-4b71-a355-fc4427be817b","Type":"ContainerStarted","Data":"90dafa35b620f86fb379751311ec620a3fd2df414d2a9a7d6b731d8b71a321b0"}
Jul 17 12:31:02.438511 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:02.438463    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xl2np" podStartSLOduration=66.19374787 podStartE2EDuration="1m10.438447733s" totalImagesPullingTime="4.244699863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:29:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:31:02.43821061 +0000 UTC m=+72.018322631" watchObservedRunningTime="2026-07-17 12:31:02.438447733 +0000 UTC m=+72.018559752"
Jul 17 12:31:02.457578 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:02.457521    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9b4b4" podStartSLOduration=65.910316261 podStartE2EDuration="1m10.457504652s" totalImagesPullingTime="4.547188391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:29:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:31:02.457376889 +0000 UTC m=+72.037488926" watchObservedRunningTime="2026-07-17 12:31:02.457504652 +0000 UTC m=+72.037616710"
Jul 17 12:31:03.219581 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:03.219531    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/efb8a146-b5cf-4b54-bb64-3b060a736ca4-metrics-tls\") pod \"dns-default-wkj54\" (UID: \"efb8a146-b5cf-4b54-bb64-3b060a736ca4\") " pod="openshift-dns/dns-default-wkj54"
Jul 17 12:31:03.219779 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:03.219687    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 12:31:03.219779 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:03.219764    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efb8a146-b5cf-4b54-bb64-3b060a736ca4-metrics-tls podName:efb8a146-b5cf-4b54-bb64-3b060a736ca4 nodeName:}" failed. No retries permitted until 2026-07-17 12:31:35.21974792 +0000 UTC m=+104.799859916 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/efb8a146-b5cf-4b54-bb64-3b060a736ca4-metrics-tls") pod "dns-default-wkj54" (UID: "efb8a146-b5cf-4b54-bb64-3b060a736ca4") : secret "dns-default-metrics-tls" not found
Jul 17 12:31:03.504833 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:03.504747    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:31:04.307124 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.307086    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-42w44"]
Jul 17 12:31:04.310263 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.310238    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-2pclj"]
Jul 17 12:31:04.310397 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.310349    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-42w44"
Jul 17 12:31:04.313075 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.312988    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-2pclj"
Jul 17 12:31:04.316507 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.316282    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 17 12:31:04.316929 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.316892    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 17 12:31:04.318782 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.318759    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 17 12:31:04.320154 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.320131    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 17 12:31:04.320274 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.320135    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-nxrxn\""
Jul 17 12:31:04.320342 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.320276    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 17 12:31:04.320488 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.320467    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 17 12:31:04.320488 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.320475    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 17 12:31:04.323089 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.323068    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-k5xsd\""
Jul 17 12:31:04.323632 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.323614    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 17 12:31:04.332618 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.332596    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 17 12:31:04.332860 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.332840    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-2pclj"]
Jul 17 12:31:04.346980 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.346953    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-42w44"]
Jul 17 12:31:04.427658 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.427632    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6652240d-b5e1-4079-b293-3e5d8469b4f9-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-2pclj\" (UID: \"6652240d-b5e1-4079-b293-3e5d8469b4f9\") " pod="openshift-insights/insights-operator-555d9d7885-2pclj"
Jul 17 12:31:04.427882 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.427668    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf5jp\" (UniqueName: \"kubernetes.io/projected/6652240d-b5e1-4079-b293-3e5d8469b4f9-kube-api-access-cf5jp\") pod \"insights-operator-555d9d7885-2pclj\" (UID: \"6652240d-b5e1-4079-b293-3e5d8469b4f9\") " pod="openshift-insights/insights-operator-555d9d7885-2pclj"
Jul 17 12:31:04.427882 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.427695    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a6b7914b-3758-4e34-bc34-309714b615c6-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-42w44\" (UID: \"a6b7914b-3758-4e34-bc34-309714b615c6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-42w44"
Jul 17 12:31:04.427882 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.427767    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a6b7914b-3758-4e34-bc34-309714b615c6-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-42w44\" (UID: \"a6b7914b-3758-4e34-bc34-309714b615c6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-42w44"
Jul 17 12:31:04.427882 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.427810    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6652240d-b5e1-4079-b293-3e5d8469b4f9-serving-cert\") pod \"insights-operator-555d9d7885-2pclj\" (UID: \"6652240d-b5e1-4079-b293-3e5d8469b4f9\") " pod="openshift-insights/insights-operator-555d9d7885-2pclj"
Jul 17 12:31:04.427882 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.427857    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttx49\" (UniqueName: \"kubernetes.io/projected/a6b7914b-3758-4e34-bc34-309714b615c6-kube-api-access-ttx49\") pod \"kube-storage-version-migrator-operator-7658d9b749-42w44\" (UID: \"a6b7914b-3758-4e34-bc34-309714b615c6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-42w44"
Jul 17 12:31:04.428089 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.427883    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6652240d-b5e1-4079-b293-3e5d8469b4f9-service-ca-bundle\") pod \"insights-operator-555d9d7885-2pclj\" (UID: \"6652240d-b5e1-4079-b293-3e5d8469b4f9\") " pod="openshift-insights/insights-operator-555d9d7885-2pclj"
Jul 17 12:31:04.428089 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.427919    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6652240d-b5e1-4079-b293-3e5d8469b4f9-tmp\") pod \"insights-operator-555d9d7885-2pclj\" (UID: \"6652240d-b5e1-4079-b293-3e5d8469b4f9\") " pod="openshift-insights/insights-operator-555d9d7885-2pclj"
Jul 17 12:31:04.428089 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.427944    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6652240d-b5e1-4079-b293-3e5d8469b4f9-snapshots\") pod \"insights-operator-555d9d7885-2pclj\" (UID: \"6652240d-b5e1-4079-b293-3e5d8469b4f9\") " pod="openshift-insights/insights-operator-555d9d7885-2pclj"
Jul 17 12:31:04.504139 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.504105    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-554b946b65-zxsp5"]
Jul 17 12:31:04.507044 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.507025    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:04.513081 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.512956    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 17 12:31:04.513382 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.513359    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 17 12:31:04.517537 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.516613    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 17 12:31:04.518021 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.517996    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5z8b4\""
Jul 17 12:31:04.522727 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.522681    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 17 12:31:04.528964 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.528944    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6652240d-b5e1-4079-b293-3e5d8469b4f9-tmp\") pod \"insights-operator-555d9d7885-2pclj\" (UID: \"6652240d-b5e1-4079-b293-3e5d8469b4f9\") " pod="openshift-insights/insights-operator-555d9d7885-2pclj"
Jul 17 12:31:04.529058 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.528976    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6652240d-b5e1-4079-b293-3e5d8469b4f9-snapshots\") pod \"insights-operator-555d9d7885-2pclj\" (UID: \"6652240d-b5e1-4079-b293-3e5d8469b4f9\") " pod="openshift-insights/insights-operator-555d9d7885-2pclj"
Jul 17 12:31:04.529058 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.529026    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6652240d-b5e1-4079-b293-3e5d8469b4f9-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-2pclj\" (UID: \"6652240d-b5e1-4079-b293-3e5d8469b4f9\") " pod="openshift-insights/insights-operator-555d9d7885-2pclj"
Jul 17 12:31:04.529168 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.529054    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf5jp\" (UniqueName: \"kubernetes.io/projected/6652240d-b5e1-4079-b293-3e5d8469b4f9-kube-api-access-cf5jp\") pod \"insights-operator-555d9d7885-2pclj\" (UID: \"6652240d-b5e1-4079-b293-3e5d8469b4f9\") " pod="openshift-insights/insights-operator-555d9d7885-2pclj"
Jul 17 12:31:04.529168 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.529080    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a6b7914b-3758-4e34-bc34-309714b615c6-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-42w44\" (UID: \"a6b7914b-3758-4e34-bc34-309714b615c6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-42w44"
Jul 17 12:31:04.529168 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.529111    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a6b7914b-3758-4e34-bc34-309714b615c6-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-42w44\" (UID: \"a6b7914b-3758-4e34-bc34-309714b615c6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-42w44"
Jul 17 12:31:04.529315 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.529251    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6652240d-b5e1-4079-b293-3e5d8469b4f9-serving-cert\") pod \"insights-operator-555d9d7885-2pclj\" (UID: \"6652240d-b5e1-4079-b293-3e5d8469b4f9\") " pod="openshift-insights/insights-operator-555d9d7885-2pclj"
Jul 17 12:31:04.529371 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.529319    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttx49\" (UniqueName: \"kubernetes.io/projected/a6b7914b-3758-4e34-bc34-309714b615c6-kube-api-access-ttx49\") pod \"kube-storage-version-migrator-operator-7658d9b749-42w44\" (UID: \"a6b7914b-3758-4e34-bc34-309714b615c6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-42w44"
Jul 17 12:31:04.529371 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.529360    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6652240d-b5e1-4079-b293-3e5d8469b4f9-service-ca-bundle\") pod \"insights-operator-555d9d7885-2pclj\" (UID: \"6652240d-b5e1-4079-b293-3e5d8469b4f9\") " pod="openshift-insights/insights-operator-555d9d7885-2pclj"
Jul 17 12:31:04.529758 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.529735    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a6b7914b-3758-4e34-bc34-309714b615c6-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-42w44\" (UID: \"a6b7914b-3758-4e34-bc34-309714b615c6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-42w44"
Jul 17 12:31:04.529917 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.529901    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6652240d-b5e1-4079-b293-3e5d8469b4f9-tmp\") pod \"insights-operator-555d9d7885-2pclj\" (UID: \"6652240d-b5e1-4079-b293-3e5d8469b4f9\") " pod="openshift-insights/insights-operator-555d9d7885-2pclj"
Jul 17 12:31:04.530170 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.530145    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6652240d-b5e1-4079-b293-3e5d8469b4f9-snapshots\") pod \"insights-operator-555d9d7885-2pclj\" (UID: \"6652240d-b5e1-4079-b293-3e5d8469b4f9\") " pod="openshift-insights/insights-operator-555d9d7885-2pclj"
Jul 17 12:31:04.530240 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.530168    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6652240d-b5e1-4079-b293-3e5d8469b4f9-service-ca-bundle\") pod \"insights-operator-555d9d7885-2pclj\" (UID: \"6652240d-b5e1-4079-b293-3e5d8469b4f9\") " pod="openshift-insights/insights-operator-555d9d7885-2pclj"
Jul 17 12:31:04.530511 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.530489    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6652240d-b5e1-4079-b293-3e5d8469b4f9-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-2pclj\" (UID: \"6652240d-b5e1-4079-b293-3e5d8469b4f9\") " pod="openshift-insights/insights-operator-555d9d7885-2pclj"
Jul 17 12:31:04.531601 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.531581    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a6b7914b-3758-4e34-bc34-309714b615c6-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-42w44\" (UID: \"a6b7914b-3758-4e34-bc34-309714b615c6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-42w44"
Jul 17 12:31:04.531691 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.531670    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6652240d-b5e1-4079-b293-3e5d8469b4f9-serving-cert\") pod \"insights-operator-555d9d7885-2pclj\" (UID: \"6652240d-b5e1-4079-b293-3e5d8469b4f9\") " pod="openshift-insights/insights-operator-555d9d7885-2pclj"
Jul 17 12:31:04.539223 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.539201    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-554b946b65-zxsp5"]
Jul 17 12:31:04.550035 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.550011    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttx49\" (UniqueName: \"kubernetes.io/projected/a6b7914b-3758-4e34-bc34-309714b615c6-kube-api-access-ttx49\") pod \"kube-storage-version-migrator-operator-7658d9b749-42w44\" (UID: \"a6b7914b-3758-4e34-bc34-309714b615c6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-42w44"
Jul 17 12:31:04.550148 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.550095    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf5jp\" (UniqueName: \"kubernetes.io/projected/6652240d-b5e1-4079-b293-3e5d8469b4f9-kube-api-access-cf5jp\") pod \"insights-operator-555d9d7885-2pclj\" (UID: \"6652240d-b5e1-4079-b293-3e5d8469b4f9\") " pod="openshift-insights/insights-operator-555d9d7885-2pclj"
Jul 17 12:31:04.623312 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.623219    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-42w44"
Jul 17 12:31:04.629332 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.629310    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-2pclj"
Jul 17 12:31:04.630585 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.630557    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfrv8\" (UniqueName: \"kubernetes.io/projected/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-kube-api-access-wfrv8\") pod \"image-registry-554b946b65-zxsp5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") " pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:04.630689 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.630596    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-image-registry-private-configuration\") pod \"image-registry-554b946b65-zxsp5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") " pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:04.630689 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.630626    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-registry-tls\") pod \"image-registry-554b946b65-zxsp5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") " pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:04.630689 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.630652    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-ca-trust-extracted\") pod \"image-registry-554b946b65-zxsp5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") " pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:04.630689 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.630667    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-trusted-ca\") pod \"image-registry-554b946b65-zxsp5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") " pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:04.630901 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.630769    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-installation-pull-secrets\") pod \"image-registry-554b946b65-zxsp5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") " pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:04.630901 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.630790    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-bound-sa-token\") pod \"image-registry-554b946b65-zxsp5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") " pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:04.630901 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.630811    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-registry-certificates\") pod \"image-registry-554b946b65-zxsp5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") " pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:04.731208 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.731173    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-image-registry-private-configuration\") pod \"image-registry-554b946b65-zxsp5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") " pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:04.731355 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.731224    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-registry-tls\") pod \"image-registry-554b946b65-zxsp5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") " pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:04.731355 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.731251    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-ca-trust-extracted\") pod \"image-registry-554b946b65-zxsp5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") " pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:04.731355 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.731274    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-trusted-ca\") pod \"image-registry-554b946b65-zxsp5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") " pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:04.731355 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.731349    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-installation-pull-secrets\") pod \"image-registry-554b946b65-zxsp5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") " pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:04.731683 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.731374    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-bound-sa-token\") pod \"image-registry-554b946b65-zxsp5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") " pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:04.731683 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.731398    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-registry-certificates\") pod \"image-registry-554b946b65-zxsp5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") " pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:04.731683 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:04.731411    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 12:31:04.731683 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:04.731437    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-554b946b65-zxsp5: secret "image-registry-tls" not found
Jul 17 12:31:04.731683 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.731453    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfrv8\" (UniqueName: \"kubernetes.io/projected/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-kube-api-access-wfrv8\") pod \"image-registry-554b946b65-zxsp5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") " pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:04.731683 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:04.731508    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-registry-tls podName:5a9b4062-a66d-4e47-83f3-269cadb6f6a5 nodeName:}" failed. No retries permitted until 2026-07-17 12:31:05.231485208 +0000 UTC m=+74.811597209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-registry-tls") pod "image-registry-554b946b65-zxsp5" (UID: "5a9b4062-a66d-4e47-83f3-269cadb6f6a5") : secret "image-registry-tls" not found
Jul 17 12:31:04.732037 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.732016    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-ca-trust-extracted\") pod \"image-registry-554b946b65-zxsp5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") " pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:04.732756 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.732730    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-registry-certificates\") pod \"image-registry-554b946b65-zxsp5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") " pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:04.733124 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.733099    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-trusted-ca\") pod \"image-registry-554b946b65-zxsp5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") " pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:04.734306 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.734284    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-image-registry-private-configuration\") pod \"image-registry-554b946b65-zxsp5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") " pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:04.734403 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.734284    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-installation-pull-secrets\") pod \"image-registry-554b946b65-zxsp5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") " pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:04.749273 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.749243    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfrv8\" (UniqueName: \"kubernetes.io/projected/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-kube-api-access-wfrv8\") pod \"image-registry-554b946b65-zxsp5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") " pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:04.754418 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.754387    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-bound-sa-token\") pod \"image-registry-554b946b65-zxsp5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") " pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:04.782097 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.782064    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-42w44"]
Jul 17 12:31:04.785065 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:31:04.785040    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6b7914b_3758_4e34_bc34_309714b615c6.slice/crio-8a17fbdedd598173d5b06e237da853ff829bdfa07fd18a7bd28503dae5d1162c WatchSource:0}: Error finding container 8a17fbdedd598173d5b06e237da853ff829bdfa07fd18a7bd28503dae5d1162c: Status 404 returned error can't find the container with id 8a17fbdedd598173d5b06e237da853ff829bdfa07fd18a7bd28503dae5d1162c
Jul 17 12:31:04.795916 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:04.795890    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-2pclj"]
Jul 17 12:31:04.798059 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:31:04.798032    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6652240d_b5e1_4079_b293_3e5d8469b4f9.slice/crio-9683ffb138b3e576cfdeb030c95252f5ff8c0f389893f0b9d3fff76b8c8d0348 WatchSource:0}: Error finding container 9683ffb138b3e576cfdeb030c95252f5ff8c0f389893f0b9d3fff76b8c8d0348: Status 404 returned error can't find the container with id 9683ffb138b3e576cfdeb030c95252f5ff8c0f389893f0b9d3fff76b8c8d0348
Jul 17 12:31:05.234667 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:05.234614    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-registry-tls\") pod \"image-registry-554b946b65-zxsp5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") " pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:05.234856 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:05.234776    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 12:31:05.234856 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:05.234797    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-554b946b65-zxsp5: secret "image-registry-tls" not found
Jul 17 12:31:05.234856 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:05.234852    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-registry-tls podName:5a9b4062-a66d-4e47-83f3-269cadb6f6a5 nodeName:}" failed. No retries permitted until 2026-07-17 12:31:06.234838662 +0000 UTC m=+75.814950662 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-registry-tls") pod "image-registry-554b946b65-zxsp5" (UID: "5a9b4062-a66d-4e47-83f3-269cadb6f6a5") : secret "image-registry-tls" not found
Jul 17 12:31:05.429485 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:05.429424    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-2pclj" event={"ID":"6652240d-b5e1-4079-b293-3e5d8469b4f9","Type":"ContainerStarted","Data":"9683ffb138b3e576cfdeb030c95252f5ff8c0f389893f0b9d3fff76b8c8d0348"}
Jul 17 12:31:05.431059 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:05.431027    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-42w44" event={"ID":"a6b7914b-3758-4e34-bc34-309714b615c6","Type":"ContainerStarted","Data":"8a17fbdedd598173d5b06e237da853ff829bdfa07fd18a7bd28503dae5d1162c"}
Jul 17 12:31:06.242007 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:06.241969    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-registry-tls\") pod \"image-registry-554b946b65-zxsp5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") " pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:06.242484 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:06.242124    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 12:31:06.242484 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:06.242142    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-554b946b65-zxsp5: secret "image-registry-tls" not found
Jul 17 12:31:06.242484 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:06.242218    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-registry-tls podName:5a9b4062-a66d-4e47-83f3-269cadb6f6a5 nodeName:}" failed. No retries permitted until 2026-07-17 12:31:08.242188005 +0000 UTC m=+77.822300006 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-registry-tls") pod "image-registry-554b946b65-zxsp5" (UID: "5a9b4062-a66d-4e47-83f3-269cadb6f6a5") : secret "image-registry-tls" not found
Jul 17 12:31:07.443176 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:07.443076    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-2pclj" event={"ID":"6652240d-b5e1-4079-b293-3e5d8469b4f9","Type":"ContainerStarted","Data":"37490c8825347400e70e6089267ec5244ae8c4807abd82722a9f13d1043ca0a6"}
Jul 17 12:31:07.444740 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:07.444710    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-42w44" event={"ID":"a6b7914b-3758-4e34-bc34-309714b615c6","Type":"ContainerStarted","Data":"be335881e2ee0c3d42c5b77f7e38eeb98637b07277fd6451b33424f556cca350"}
Jul 17 12:31:07.466953 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:07.466900    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-2pclj" podStartSLOduration=1.187047265 podStartE2EDuration="3.466882188s" totalImagesPullingTime="2.279834923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:31:07.465639741 +0000 UTC m=+77.045751758" watchObservedRunningTime="2026-07-17 12:31:07.466882188 +0000 UTC m=+77.046994206"
Jul 17 12:31:07.487114 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:07.487048    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-42w44" podStartSLOduration=1.191196425 podStartE2EDuration="3.487030057s" totalImagesPullingTime="2.295833632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:31:07.486423936 +0000 UTC m=+77.066536010" watchObservedRunningTime="2026-07-17 12:31:07.487030057 +0000 UTC m=+77.067142075"
Jul 17 12:31:08.060933 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:08.060901    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xmzkp"]
Jul 17 12:31:08.064940 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:08.064908    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xmzkp"
Jul 17 12:31:08.067757 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:08.067732    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-7sr2v\""
Jul 17 12:31:08.068872 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:08.068854    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 17 12:31:08.068965 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:08.068879    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 17 12:31:08.076061 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:08.076041    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xmzkp"]
Jul 17 12:31:08.156972 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:08.156918    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45v6l\" (UniqueName: \"kubernetes.io/projected/b937bdf9-4b6d-4fd0-aa58-7e721cdd064e-kube-api-access-45v6l\") pod \"migrator-7cc9c7ffd7-xmzkp\" (UID: \"b937bdf9-4b6d-4fd0-aa58-7e721cdd064e\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xmzkp"
Jul 17 12:31:08.257726 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:08.257669    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-registry-tls\") pod \"image-registry-554b946b65-zxsp5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") " pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:08.257850 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:08.257751    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45v6l\" (UniqueName: \"kubernetes.io/projected/b937bdf9-4b6d-4fd0-aa58-7e721cdd064e-kube-api-access-45v6l\") pod \"migrator-7cc9c7ffd7-xmzkp\" (UID: \"b937bdf9-4b6d-4fd0-aa58-7e721cdd064e\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xmzkp"
Jul 17 12:31:08.257850 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:08.257816    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 12:31:08.257850 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:08.257839    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-554b946b65-zxsp5: secret "image-registry-tls" not found
Jul 17 12:31:08.257947 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:08.257900    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-registry-tls podName:5a9b4062-a66d-4e47-83f3-269cadb6f6a5 nodeName:}" failed. No retries permitted until 2026-07-17 12:31:12.257884288 +0000 UTC m=+81.837996285 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-registry-tls") pod "image-registry-554b946b65-zxsp5" (UID: "5a9b4062-a66d-4e47-83f3-269cadb6f6a5") : secret "image-registry-tls" not found
Jul 17 12:31:08.266171 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:08.266140    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45v6l\" (UniqueName: \"kubernetes.io/projected/b937bdf9-4b6d-4fd0-aa58-7e721cdd064e-kube-api-access-45v6l\") pod \"migrator-7cc9c7ffd7-xmzkp\" (UID: \"b937bdf9-4b6d-4fd0-aa58-7e721cdd064e\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xmzkp"
Jul 17 12:31:08.373832 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:08.373742    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xmzkp"
Jul 17 12:31:08.509130 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:08.509073    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xmzkp"]
Jul 17 12:31:08.511628 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:31:08.511590    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb937bdf9_4b6d_4fd0_aa58_7e721cdd064e.slice/crio-6fdf830a9be71c8726a0c7e2473e81bdf23c3778c0aecd5bf6cc164b019b5e5a WatchSource:0}: Error finding container 6fdf830a9be71c8726a0c7e2473e81bdf23c3778c0aecd5bf6cc164b019b5e5a: Status 404 returned error can't find the container with id 6fdf830a9be71c8726a0c7e2473e81bdf23c3778c0aecd5bf6cc164b019b5e5a
Jul 17 12:31:09.062223 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:09.062188    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vd7js"]
Jul 17 12:31:09.067839 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:09.067813    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vd7js"
Jul 17 12:31:09.070383 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:09.070357    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 17 12:31:09.070598 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:09.070580    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 17 12:31:09.071562 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:09.071547    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sqn4d\""
Jul 17 12:31:09.077123 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:09.077101    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vd7js"]
Jul 17 12:31:09.165542 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:09.165498    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9fe27e83-0d04-4f44-9b37-f410e08e5f51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vd7js\" (UID: \"9fe27e83-0d04-4f44-9b37-f410e08e5f51\") " pod="openshift-insights/insights-runtime-extractor-vd7js"
Jul 17 12:31:09.165749 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:09.165556    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9fe27e83-0d04-4f44-9b37-f410e08e5f51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vd7js\" (UID: \"9fe27e83-0d04-4f44-9b37-f410e08e5f51\") " pod="openshift-insights/insights-runtime-extractor-vd7js"
Jul 17 12:31:09.165749 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:09.165654    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9fe27e83-0d04-4f44-9b37-f410e08e5f51-crio-socket\") pod \"insights-runtime-extractor-vd7js\" (UID: \"9fe27e83-0d04-4f44-9b37-f410e08e5f51\") " pod="openshift-insights/insights-runtime-extractor-vd7js"
Jul 17 12:31:09.165863 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:09.165789    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkgzd\" (UniqueName: \"kubernetes.io/projected/9fe27e83-0d04-4f44-9b37-f410e08e5f51-kube-api-access-fkgzd\") pod \"insights-runtime-extractor-vd7js\" (UID: \"9fe27e83-0d04-4f44-9b37-f410e08e5f51\") " pod="openshift-insights/insights-runtime-extractor-vd7js"
Jul 17 12:31:09.165921 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:09.165863    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9fe27e83-0d04-4f44-9b37-f410e08e5f51-data-volume\") pod \"insights-runtime-extractor-vd7js\" (UID: \"9fe27e83-0d04-4f44-9b37-f410e08e5f51\") " pod="openshift-insights/insights-runtime-extractor-vd7js"
Jul 17 12:31:09.267174 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:09.267117    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9fe27e83-0d04-4f44-9b37-f410e08e5f51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vd7js\" (UID: \"9fe27e83-0d04-4f44-9b37-f410e08e5f51\") " pod="openshift-insights/insights-runtime-extractor-vd7js"
Jul 17 12:31:09.267357 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:09.267194    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9fe27e83-0d04-4f44-9b37-f410e08e5f51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vd7js\" (UID: \"9fe27e83-0d04-4f44-9b37-f410e08e5f51\") " pod="openshift-insights/insights-runtime-extractor-vd7js"
Jul 17 12:31:09.267357 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:09.267216    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9fe27e83-0d04-4f44-9b37-f410e08e5f51-crio-socket\") pod \"insights-runtime-extractor-vd7js\" (UID: \"9fe27e83-0d04-4f44-9b37-f410e08e5f51\") " pod="openshift-insights/insights-runtime-extractor-vd7js"
Jul 17 12:31:09.267357 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:09.267241    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 12:31:09.267357 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:09.267270    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkgzd\" (UniqueName: \"kubernetes.io/projected/9fe27e83-0d04-4f44-9b37-f410e08e5f51-kube-api-access-fkgzd\") pod \"insights-runtime-extractor-vd7js\" (UID: \"9fe27e83-0d04-4f44-9b37-f410e08e5f51\") " pod="openshift-insights/insights-runtime-extractor-vd7js"
Jul 17 12:31:09.267357 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:09.267330    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9fe27e83-0d04-4f44-9b37-f410e08e5f51-kube-rbac-proxy-cm podName:9fe27e83-0d04-4f44-9b37-f410e08e5f51 nodeName:}" failed. No retries permitted until 2026-07-17 12:31:09.767305337 +0000 UTC m=+79.347417334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9fe27e83-0d04-4f44-9b37-f410e08e5f51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vd7js" (UID: "9fe27e83-0d04-4f44-9b37-f410e08e5f51") : configmap "kube-rbac-proxy" not found
Jul 17 12:31:09.267357 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:09.267349    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9fe27e83-0d04-4f44-9b37-f410e08e5f51-crio-socket\") pod \"insights-runtime-extractor-vd7js\" (UID: \"9fe27e83-0d04-4f44-9b37-f410e08e5f51\") " pod="openshift-insights/insights-runtime-extractor-vd7js"
Jul 17 12:31:09.267653 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:09.267370    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 12:31:09.267653 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:09.267404    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9fe27e83-0d04-4f44-9b37-f410e08e5f51-data-volume\") pod \"insights-runtime-extractor-vd7js\" (UID: \"9fe27e83-0d04-4f44-9b37-f410e08e5f51\") " pod="openshift-insights/insights-runtime-extractor-vd7js"
Jul 17 12:31:09.267653 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:09.267441    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fe27e83-0d04-4f44-9b37-f410e08e5f51-insights-runtime-extractor-tls podName:9fe27e83-0d04-4f44-9b37-f410e08e5f51 nodeName:}" failed. No retries permitted until 2026-07-17 12:31:09.767423753 +0000 UTC m=+79.347535751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9fe27e83-0d04-4f44-9b37-f410e08e5f51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vd7js" (UID: "9fe27e83-0d04-4f44-9b37-f410e08e5f51") : secret "insights-runtime-extractor-tls" not found
Jul 17 12:31:09.267790 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:09.267772    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9fe27e83-0d04-4f44-9b37-f410e08e5f51-data-volume\") pod \"insights-runtime-extractor-vd7js\" (UID: \"9fe27e83-0d04-4f44-9b37-f410e08e5f51\") " pod="openshift-insights/insights-runtime-extractor-vd7js"
Jul 17 12:31:09.279063 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:09.279031    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkgzd\" (UniqueName: \"kubernetes.io/projected/9fe27e83-0d04-4f44-9b37-f410e08e5f51-kube-api-access-fkgzd\") pod \"insights-runtime-extractor-vd7js\" (UID: \"9fe27e83-0d04-4f44-9b37-f410e08e5f51\") " pod="openshift-insights/insights-runtime-extractor-vd7js"
Jul 17 12:31:09.458573 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:09.458486    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xmzkp" event={"ID":"b937bdf9-4b6d-4fd0-aa58-7e721cdd064e","Type":"ContainerStarted","Data":"6fdf830a9be71c8726a0c7e2473e81bdf23c3778c0aecd5bf6cc164b019b5e5a"}
Jul 17 12:31:09.772056 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:09.772031    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9fe27e83-0d04-4f44-9b37-f410e08e5f51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vd7js\" (UID: \"9fe27e83-0d04-4f44-9b37-f410e08e5f51\") " pod="openshift-insights/insights-runtime-extractor-vd7js"
Jul 17 12:31:09.772351 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:09.772071    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9fe27e83-0d04-4f44-9b37-f410e08e5f51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vd7js\" (UID: \"9fe27e83-0d04-4f44-9b37-f410e08e5f51\") " pod="openshift-insights/insights-runtime-extractor-vd7js"
Jul 17 12:31:09.772351 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:09.772173    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 12:31:09.772351 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:09.772189    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 12:31:09.772351 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:09.772253    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9fe27e83-0d04-4f44-9b37-f410e08e5f51-kube-rbac-proxy-cm podName:9fe27e83-0d04-4f44-9b37-f410e08e5f51 nodeName:}" failed. No retries permitted until 2026-07-17 12:31:10.77223255 +0000 UTC m=+80.352344549 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9fe27e83-0d04-4f44-9b37-f410e08e5f51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vd7js" (UID: "9fe27e83-0d04-4f44-9b37-f410e08e5f51") : configmap "kube-rbac-proxy" not found
Jul 17 12:31:09.772351 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:09.772270    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fe27e83-0d04-4f44-9b37-f410e08e5f51-insights-runtime-extractor-tls podName:9fe27e83-0d04-4f44-9b37-f410e08e5f51 nodeName:}" failed. No retries permitted until 2026-07-17 12:31:10.772262315 +0000 UTC m=+80.352374311 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9fe27e83-0d04-4f44-9b37-f410e08e5f51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vd7js" (UID: "9fe27e83-0d04-4f44-9b37-f410e08e5f51") : secret "insights-runtime-extractor-tls" not found
Jul 17 12:31:10.463441 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:10.463404    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xmzkp" event={"ID":"b937bdf9-4b6d-4fd0-aa58-7e721cdd064e","Type":"ContainerStarted","Data":"a1812cab587c39b2c7f931744f11eba40ea92a2376d3ed3e630a31b124acf54d"}
Jul 17 12:31:10.463614 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:10.463453    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xmzkp" event={"ID":"b937bdf9-4b6d-4fd0-aa58-7e721cdd064e","Type":"ContainerStarted","Data":"1306f4e5c8b5b2b2475d4c667dd3cc6417e8813dac1e89298eeedfaf86088fa4"}
Jul 17 12:31:10.483598 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:10.483543    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xmzkp" podStartSLOduration=1.275813576 podStartE2EDuration="2.483528732s" totalImagesPullingTime="1.207715156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:31:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:31:10.481838358 +0000 UTC m=+80.061950403" watchObservedRunningTime="2026-07-17 12:31:10.483528732 +0000 UTC m=+80.063640751"
Jul 17 12:31:10.781018 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:10.780988    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9fe27e83-0d04-4f44-9b37-f410e08e5f51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vd7js\" (UID: \"9fe27e83-0d04-4f44-9b37-f410e08e5f51\") " pod="openshift-insights/insights-runtime-extractor-vd7js"
Jul 17 12:31:10.781468 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:10.781033    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9fe27e83-0d04-4f44-9b37-f410e08e5f51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vd7js\" (UID: \"9fe27e83-0d04-4f44-9b37-f410e08e5f51\") " pod="openshift-insights/insights-runtime-extractor-vd7js"
Jul 17 12:31:10.781468 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:10.781121    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 12:31:10.781468 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:10.781138    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 12:31:10.781468 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:10.781189    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fe27e83-0d04-4f44-9b37-f410e08e5f51-insights-runtime-extractor-tls podName:9fe27e83-0d04-4f44-9b37-f410e08e5f51 nodeName:}" failed. No retries permitted until 2026-07-17 12:31:12.781176173 +0000 UTC m=+82.361288170 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9fe27e83-0d04-4f44-9b37-f410e08e5f51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vd7js" (UID: "9fe27e83-0d04-4f44-9b37-f410e08e5f51") : secret "insights-runtime-extractor-tls" not found
Jul 17 12:31:10.781468 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:10.781230    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9fe27e83-0d04-4f44-9b37-f410e08e5f51-kube-rbac-proxy-cm podName:9fe27e83-0d04-4f44-9b37-f410e08e5f51 nodeName:}" failed. No retries permitted until 2026-07-17 12:31:12.781223077 +0000 UTC m=+82.361335073 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9fe27e83-0d04-4f44-9b37-f410e08e5f51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vd7js" (UID: "9fe27e83-0d04-4f44-9b37-f410e08e5f51") : configmap "kube-rbac-proxy" not found
Jul 17 12:31:11.843229 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:11.843204    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g8rzg_688df482-0ac3-4a72-9632-c6a8c18b0b8d/dns-node-resolver/0.log"
Jul 17 12:31:12.291593 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:12.291560    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-registry-tls\") pod \"image-registry-554b946b65-zxsp5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") " pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:12.291787 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:12.291725    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 12:31:12.291787 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:12.291743    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-554b946b65-zxsp5: secret "image-registry-tls" not found
Jul 17 12:31:12.291861 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:12.291797    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-registry-tls podName:5a9b4062-a66d-4e47-83f3-269cadb6f6a5 nodeName:}" failed. No retries permitted until 2026-07-17 12:31:20.291782698 +0000 UTC m=+89.871894703 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-registry-tls") pod "image-registry-554b946b65-zxsp5" (UID: "5a9b4062-a66d-4e47-83f3-269cadb6f6a5") : secret "image-registry-tls" not found
Jul 17 12:31:12.795862 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:12.795828    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9fe27e83-0d04-4f44-9b37-f410e08e5f51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vd7js\" (UID: \"9fe27e83-0d04-4f44-9b37-f410e08e5f51\") " pod="openshift-insights/insights-runtime-extractor-vd7js"
Jul 17 12:31:12.796058 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:12.795872    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9fe27e83-0d04-4f44-9b37-f410e08e5f51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vd7js\" (UID: \"9fe27e83-0d04-4f44-9b37-f410e08e5f51\") " pod="openshift-insights/insights-runtime-extractor-vd7js"
Jul 17 12:31:12.796058 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:12.795974    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 12:31:12.796173 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:12.796059    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9fe27e83-0d04-4f44-9b37-f410e08e5f51-kube-rbac-proxy-cm podName:9fe27e83-0d04-4f44-9b37-f410e08e5f51 nodeName:}" failed. No retries permitted until 2026-07-17 12:31:16.796038746 +0000 UTC m=+86.376150755 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9fe27e83-0d04-4f44-9b37-f410e08e5f51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vd7js" (UID: "9fe27e83-0d04-4f44-9b37-f410e08e5f51") : configmap "kube-rbac-proxy" not found
Jul 17 12:31:12.796173 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:12.795988    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 12:31:12.796173 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:12.796119    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fe27e83-0d04-4f44-9b37-f410e08e5f51-insights-runtime-extractor-tls podName:9fe27e83-0d04-4f44-9b37-f410e08e5f51 nodeName:}" failed. No retries permitted until 2026-07-17 12:31:16.796106758 +0000 UTC m=+86.376218778 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9fe27e83-0d04-4f44-9b37-f410e08e5f51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vd7js" (UID: "9fe27e83-0d04-4f44-9b37-f410e08e5f51") : secret "insights-runtime-extractor-tls" not found
Jul 17 12:31:12.843235 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:12.843204    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xxld5_cc19f981-8855-47ce-98c7-6e8e9c0e65d2/node-ca/0.log"
Jul 17 12:31:13.443869 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:13.443837    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4h4tb_ed27f8cc-6874-4594-905b-2483d9f9463d/serve-healthcheck-canary/0.log"
Jul 17 12:31:14.043650 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:14.043623    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-xmzkp_b937bdf9-4b6d-4fd0-aa58-7e721cdd064e/migrator/0.log"
Jul 17 12:31:14.257049 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:14.257019    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-xmzkp_b937bdf9-4b6d-4fd0-aa58-7e721cdd064e/graceful-termination/0.log"
Jul 17 12:31:14.454892 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:14.454798    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-42w44_a6b7914b-3758-4e34-bc34-309714b615c6/kube-storage-version-migrator-operator/0.log"
Jul 17 12:31:16.827990 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:16.827956    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9fe27e83-0d04-4f44-9b37-f410e08e5f51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vd7js\" (UID: \"9fe27e83-0d04-4f44-9b37-f410e08e5f51\") " pod="openshift-insights/insights-runtime-extractor-vd7js"
Jul 17 12:31:16.828369 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:16.828037    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9fe27e83-0d04-4f44-9b37-f410e08e5f51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vd7js\" (UID: \"9fe27e83-0d04-4f44-9b37-f410e08e5f51\") " pod="openshift-insights/insights-runtime-extractor-vd7js"
Jul 17 12:31:16.828369 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:16.828106    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 12:31:16.828369 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:16.828125    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 12:31:16.828369 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:16.828169    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9fe27e83-0d04-4f44-9b37-f410e08e5f51-kube-rbac-proxy-cm podName:9fe27e83-0d04-4f44-9b37-f410e08e5f51 nodeName:}" failed. No retries permitted until 2026-07-17 12:31:24.828154632 +0000 UTC m=+94.408266628 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9fe27e83-0d04-4f44-9b37-f410e08e5f51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vd7js" (UID: "9fe27e83-0d04-4f44-9b37-f410e08e5f51") : configmap "kube-rbac-proxy" not found
Jul 17 12:31:16.828369 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:16.828184    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fe27e83-0d04-4f44-9b37-f410e08e5f51-insights-runtime-extractor-tls podName:9fe27e83-0d04-4f44-9b37-f410e08e5f51 nodeName:}" failed. No retries permitted until 2026-07-17 12:31:24.828176542 +0000 UTC m=+94.408288538 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9fe27e83-0d04-4f44-9b37-f410e08e5f51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vd7js" (UID: "9fe27e83-0d04-4f44-9b37-f410e08e5f51") : secret "insights-runtime-extractor-tls" not found
Jul 17 12:31:17.814772 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:17.814739    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-554b946b65-zxsp5"]
Jul 17 12:31:17.815019 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:17.814999    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-554b946b65-zxsp5" podUID="5a9b4062-a66d-4e47-83f3-269cadb6f6a5"
Jul 17 12:31:17.846578 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:17.846540    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79777dbb55-7fclv"]
Jul 17 12:31:17.849630 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:17.849613    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:31:17.862229 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:17.862204    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79777dbb55-7fclv"]
Jul 17 12:31:17.936789 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:17.936753    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e4764c4-743b-4010-a3f1-57b54e8b80a5-trusted-ca\") pod \"image-registry-79777dbb55-7fclv\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") " pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:31:17.936789 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:17.936789    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e4764c4-743b-4010-a3f1-57b54e8b80a5-bound-sa-token\") pod \"image-registry-79777dbb55-7fclv\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") " pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:31:17.936997 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:17.936823    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e4764c4-743b-4010-a3f1-57b54e8b80a5-installation-pull-secrets\") pod \"image-registry-79777dbb55-7fclv\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") " pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:31:17.936997 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:17.936873    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-729ww\" (UniqueName: \"kubernetes.io/projected/4e4764c4-743b-4010-a3f1-57b54e8b80a5-kube-api-access-729ww\") pod \"image-registry-79777dbb55-7fclv\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") " pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:31:17.936997 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:17.936939    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e4764c4-743b-4010-a3f1-57b54e8b80a5-registry-tls\") pod \"image-registry-79777dbb55-7fclv\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") " pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:31:17.936997 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:17.936962    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e4764c4-743b-4010-a3f1-57b54e8b80a5-image-registry-private-configuration\") pod \"image-registry-79777dbb55-7fclv\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") " pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:31:17.937136 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:17.937032    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e4764c4-743b-4010-a3f1-57b54e8b80a5-ca-trust-extracted\") pod \"image-registry-79777dbb55-7fclv\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") " pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:31:17.937136 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:17.937067    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e4764c4-743b-4010-a3f1-57b54e8b80a5-registry-certificates\") pod \"image-registry-79777dbb55-7fclv\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") " pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:31:18.037943 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.037901    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e4764c4-743b-4010-a3f1-57b54e8b80a5-image-registry-private-configuration\") pod \"image-registry-79777dbb55-7fclv\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") " pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:31:18.038111 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.037973    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e4764c4-743b-4010-a3f1-57b54e8b80a5-ca-trust-extracted\") pod \"image-registry-79777dbb55-7fclv\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") " pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:31:18.038111 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.038002    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e4764c4-743b-4010-a3f1-57b54e8b80a5-registry-certificates\") pod \"image-registry-79777dbb55-7fclv\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") " pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:31:18.038206 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.038155    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e4764c4-743b-4010-a3f1-57b54e8b80a5-trusted-ca\") pod \"image-registry-79777dbb55-7fclv\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") " pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:31:18.038206 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.038181    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e4764c4-743b-4010-a3f1-57b54e8b80a5-bound-sa-token\") pod \"image-registry-79777dbb55-7fclv\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") " pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:31:18.038297 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.038213    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e4764c4-743b-4010-a3f1-57b54e8b80a5-installation-pull-secrets\") pod \"image-registry-79777dbb55-7fclv\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") " pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:31:18.038297 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.038230    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-729ww\" (UniqueName: \"kubernetes.io/projected/4e4764c4-743b-4010-a3f1-57b54e8b80a5-kube-api-access-729ww\") pod \"image-registry-79777dbb55-7fclv\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") " pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:31:18.038621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.038586    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e4764c4-743b-4010-a3f1-57b54e8b80a5-registry-tls\") pod \"image-registry-79777dbb55-7fclv\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") " pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:31:18.038913 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.038888    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e4764c4-743b-4010-a3f1-57b54e8b80a5-ca-trust-extracted\") pod \"image-registry-79777dbb55-7fclv\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") " pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:31:18.039356 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.039331    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e4764c4-743b-4010-a3f1-57b54e8b80a5-registry-certificates\") pod \"image-registry-79777dbb55-7fclv\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") " pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:31:18.043089 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.040878    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e4764c4-743b-4010-a3f1-57b54e8b80a5-trusted-ca\") pod \"image-registry-79777dbb55-7fclv\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") " pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:31:18.043089 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.041352    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e4764c4-743b-4010-a3f1-57b54e8b80a5-image-registry-private-configuration\") pod \"image-registry-79777dbb55-7fclv\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") " pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:31:18.043089 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.041909    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e4764c4-743b-4010-a3f1-57b54e8b80a5-installation-pull-secrets\") pod \"image-registry-79777dbb55-7fclv\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") " pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:31:18.044339 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.044320    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e4764c4-743b-4010-a3f1-57b54e8b80a5-registry-tls\") pod \"image-registry-79777dbb55-7fclv\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") " pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:31:18.052855 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.052831    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e4764c4-743b-4010-a3f1-57b54e8b80a5-bound-sa-token\") pod \"image-registry-79777dbb55-7fclv\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") " pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:31:18.053308 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.053290    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-729ww\" (UniqueName: \"kubernetes.io/projected/4e4764c4-743b-4010-a3f1-57b54e8b80a5-kube-api-access-729ww\") pod \"image-registry-79777dbb55-7fclv\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") " pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:31:18.158733 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.158615    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:31:18.290203 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.290170    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79777dbb55-7fclv"]
Jul 17 12:31:18.293221 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:31:18.293197    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4764c4_743b_4010_a3f1_57b54e8b80a5.slice/crio-fcb2d3101ab6c4f398664528ac39599166558ce8709d177c53c4c3237b9e41ec WatchSource:0}: Error finding container fcb2d3101ab6c4f398664528ac39599166558ce8709d177c53c4c3237b9e41ec: Status 404 returned error can't find the container with id fcb2d3101ab6c4f398664528ac39599166558ce8709d177c53c4c3237b9e41ec
Jul 17 12:31:18.487880 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.487789    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79777dbb55-7fclv" event={"ID":"4e4764c4-743b-4010-a3f1-57b54e8b80a5","Type":"ContainerStarted","Data":"ef3c43920f78334dbf5258c279918b7d79863834d9632d51480164bc2c57b924"}
Jul 17 12:31:18.487880 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.487815    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:18.487880 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.487843    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79777dbb55-7fclv" event={"ID":"4e4764c4-743b-4010-a3f1-57b54e8b80a5","Type":"ContainerStarted","Data":"fcb2d3101ab6c4f398664528ac39599166558ce8709d177c53c4c3237b9e41ec"}
Jul 17 12:31:18.494938 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.494906    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:18.532762 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.532694    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-79777dbb55-7fclv" podStartSLOduration=1.5326768579999999 podStartE2EDuration="1.532676858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:31:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:31:18.531688342 +0000 UTC m=+88.111800387" watchObservedRunningTime="2026-07-17 12:31:18.532676858 +0000 UTC m=+88.112788878"
Jul 17 12:31:18.543057 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.542992    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-registry-certificates\") pod \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") "
Jul 17 12:31:18.543208 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.543085    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfrv8\" (UniqueName: \"kubernetes.io/projected/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-kube-api-access-wfrv8\") pod \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") "
Jul 17 12:31:18.543208 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.543110    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-ca-trust-extracted\") pod \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") "
Jul 17 12:31:18.543208 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.543140    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-trusted-ca\") pod \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") "
Jul 17 12:31:18.543208 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.543158    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-bound-sa-token\") pod \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") "
Jul 17 12:31:18.543208 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.543178    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-installation-pull-secrets\") pod \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") "
Jul 17 12:31:18.543455 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.543211    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-image-registry-private-configuration\") pod \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\" (UID: \"5a9b4062-a66d-4e47-83f3-269cadb6f6a5\") "
Jul 17 12:31:18.543508 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.543460    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5a9b4062-a66d-4e47-83f3-269cadb6f6a5" (UID: "5a9b4062-a66d-4e47-83f3-269cadb6f6a5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:31:18.544097 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.544072    2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-ca-trust-extracted\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:31:18.544681 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.544643    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5a9b4062-a66d-4e47-83f3-269cadb6f6a5" (UID: "5a9b4062-a66d-4e47-83f3-269cadb6f6a5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:31:18.544862 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.544844    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5a9b4062-a66d-4e47-83f3-269cadb6f6a5" (UID: "5a9b4062-a66d-4e47-83f3-269cadb6f6a5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:31:18.545552 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.545527    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-kube-api-access-wfrv8" (OuterVolumeSpecName: "kube-api-access-wfrv8") pod "5a9b4062-a66d-4e47-83f3-269cadb6f6a5" (UID: "5a9b4062-a66d-4e47-83f3-269cadb6f6a5"). InnerVolumeSpecName "kube-api-access-wfrv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:31:18.545734 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.545691    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5a9b4062-a66d-4e47-83f3-269cadb6f6a5" (UID: "5a9b4062-a66d-4e47-83f3-269cadb6f6a5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:31:18.545734 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.545717    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5a9b4062-a66d-4e47-83f3-269cadb6f6a5" (UID: "5a9b4062-a66d-4e47-83f3-269cadb6f6a5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:31:18.546228 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.546210    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5a9b4062-a66d-4e47-83f3-269cadb6f6a5" (UID: "5a9b4062-a66d-4e47-83f3-269cadb6f6a5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:31:18.644496 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.644461    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfrv8\" (UniqueName: \"kubernetes.io/projected/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-kube-api-access-wfrv8\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:31:18.644496 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.644489    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-trusted-ca\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:31:18.644496 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.644498    2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-bound-sa-token\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:31:18.644496 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.644508    2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-installation-pull-secrets\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:31:18.644761 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.644517    2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-image-registry-private-configuration\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:31:18.644761 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:18.644526    2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-registry-certificates\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:31:19.490530 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:19.490498    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-554b946b65-zxsp5"
Jul 17 12:31:19.490982 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:19.490646    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:31:19.524897 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:19.524864    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-554b946b65-zxsp5"]
Jul 17 12:31:19.529613 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:19.529586    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-554b946b65-zxsp5"]
Jul 17 12:31:19.652547 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:19.652508    2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a9b4062-a66d-4e47-83f3-269cadb6f6a5-registry-tls\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:31:21.012424 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:21.012393    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a9b4062-a66d-4e47-83f3-269cadb6f6a5" path="/var/lib/kubelet/pods/5a9b4062-a66d-4e47-83f3-269cadb6f6a5/volumes"
Jul 17 12:31:24.899321 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:24.899278    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9fe27e83-0d04-4f44-9b37-f410e08e5f51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vd7js\" (UID: \"9fe27e83-0d04-4f44-9b37-f410e08e5f51\") " pod="openshift-insights/insights-runtime-extractor-vd7js"
Jul 17 12:31:24.899321 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:24.899328    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9fe27e83-0d04-4f44-9b37-f410e08e5f51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vd7js\" (UID: \"9fe27e83-0d04-4f44-9b37-f410e08e5f51\") " pod="openshift-insights/insights-runtime-extractor-vd7js"
Jul 17 12:31:24.899849 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:24.899828    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9fe27e83-0d04-4f44-9b37-f410e08e5f51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vd7js\" (UID: \"9fe27e83-0d04-4f44-9b37-f410e08e5f51\") " pod="openshift-insights/insights-runtime-extractor-vd7js"
Jul 17 12:31:24.901622 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:24.901599    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9fe27e83-0d04-4f44-9b37-f410e08e5f51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vd7js\" (UID: \"9fe27e83-0d04-4f44-9b37-f410e08e5f51\") " pod="openshift-insights/insights-runtime-extractor-vd7js"
Jul 17 12:31:24.978122 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:24.978063    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vd7js"
Jul 17 12:31:25.115387 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:25.115360    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vd7js"]
Jul 17 12:31:25.118013 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:31:25.117980    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe27e83_0d04_4f44_9b37_f410e08e5f51.slice/crio-87153824f44acc2dafcc2ef7bad6ac62fd9c0b49e098508840b06602cf84e80b WatchSource:0}: Error finding container 87153824f44acc2dafcc2ef7bad6ac62fd9c0b49e098508840b06602cf84e80b: Status 404 returned error can't find the container with id 87153824f44acc2dafcc2ef7bad6ac62fd9c0b49e098508840b06602cf84e80b
Jul 17 12:31:25.517265 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:25.517230    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vd7js" event={"ID":"9fe27e83-0d04-4f44-9b37-f410e08e5f51","Type":"ContainerStarted","Data":"87153824f44acc2dafcc2ef7bad6ac62fd9c0b49e098508840b06602cf84e80b"}
Jul 17 12:31:30.534918 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:30.534873    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vd7js" event={"ID":"9fe27e83-0d04-4f44-9b37-f410e08e5f51","Type":"ContainerStarted","Data":"1b8fd88a75adb255e50b12e6896925dc7d79c5a7435415261d33667153156b63"}
Jul 17 12:31:33.424487 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.424455    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xl2np"
Jul 17 12:31:33.839845 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.839808    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-9dvjl"]
Jul 17 12:31:33.850893 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.850864    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6jrn7"]
Jul 17 12:31:33.851047 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.851006    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-9dvjl"
Jul 17 12:31:33.853818 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.853758    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 17 12:31:33.854018 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.853995    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-777bp\""
Jul 17 12:31:33.854147 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.854018    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 17 12:31:33.854147 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.854034    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 17 12:31:33.854492 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.854463    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 17 12:31:33.854609 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.854516    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6jrn7"
Jul 17 12:31:33.854609 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.854548    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 17 12:31:33.854763 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.854620    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 17 12:31:33.857277 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.857258    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 17 12:31:33.857469 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.857395    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zxvjl\""
Jul 17 12:31:33.857796 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.857503    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 17 12:31:33.857796 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.857516    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 17 12:31:33.864638 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.864178    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-9dvjl"]
Jul 17 12:31:33.977859 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.977824    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b9cd22df-55bb-4bfb-8108-8bbad86b57cd-root\") pod \"node-exporter-6jrn7\" (UID: \"b9cd22df-55bb-4bfb-8108-8bbad86b57cd\") " pod="openshift-monitoring/node-exporter-6jrn7"
Jul 17 12:31:33.977859 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.977861    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/45cbefc2-d0c1-41ac-a78e-be14100134c4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-9dvjl\" (UID: \"45cbefc2-d0c1-41ac-a78e-be14100134c4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-9dvjl"
Jul 17 12:31:33.978109 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.977883    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-644fv\" (UniqueName: \"kubernetes.io/projected/45cbefc2-d0c1-41ac-a78e-be14100134c4-kube-api-access-644fv\") pod \"kube-state-metrics-56c5594674-9dvjl\" (UID: \"45cbefc2-d0c1-41ac-a78e-be14100134c4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-9dvjl"
Jul 17 12:31:33.978109 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.977979    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b9cd22df-55bb-4bfb-8108-8bbad86b57cd-node-exporter-textfile\") pod \"node-exporter-6jrn7\" (UID: \"b9cd22df-55bb-4bfb-8108-8bbad86b57cd\") " pod="openshift-monitoring/node-exporter-6jrn7"
Jul 17 12:31:33.978109 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.978029    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b9cd22df-55bb-4bfb-8108-8bbad86b57cd-node-exporter-accelerators-collector-config\") pod \"node-exporter-6jrn7\" (UID: \"b9cd22df-55bb-4bfb-8108-8bbad86b57cd\") " pod="openshift-monitoring/node-exporter-6jrn7"
Jul 17 12:31:33.978109 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.978056    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/45cbefc2-d0c1-41ac-a78e-be14100134c4-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-9dvjl\" (UID: \"45cbefc2-d0c1-41ac-a78e-be14100134c4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-9dvjl"
Jul 17 12:31:33.978109 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.978092    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9cd22df-55bb-4bfb-8108-8bbad86b57cd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6jrn7\" (UID: \"b9cd22df-55bb-4bfb-8108-8bbad86b57cd\") " pod="openshift-monitoring/node-exporter-6jrn7"
Jul 17 12:31:33.978348 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.978161    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45cbefc2-d0c1-41ac-a78e-be14100134c4-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-9dvjl\" (UID: \"45cbefc2-d0c1-41ac-a78e-be14100134c4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-9dvjl"
Jul 17 12:31:33.978348 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.978218    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnfp5\" (UniqueName: \"kubernetes.io/projected/b9cd22df-55bb-4bfb-8108-8bbad86b57cd-kube-api-access-dnfp5\") pod \"node-exporter-6jrn7\" (UID: \"b9cd22df-55bb-4bfb-8108-8bbad86b57cd\") " pod="openshift-monitoring/node-exporter-6jrn7"
Jul 17 12:31:33.978348 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.978264    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9cd22df-55bb-4bfb-8108-8bbad86b57cd-sys\") pod \"node-exporter-6jrn7\" (UID: \"b9cd22df-55bb-4bfb-8108-8bbad86b57cd\") " pod="openshift-monitoring/node-exporter-6jrn7"
Jul 17 12:31:33.978348 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.978306    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b9cd22df-55bb-4bfb-8108-8bbad86b57cd-node-exporter-tls\") pod \"node-exporter-6jrn7\" (UID: \"b9cd22df-55bb-4bfb-8108-8bbad86b57cd\") " pod="openshift-monitoring/node-exporter-6jrn7"
Jul 17 12:31:33.978348 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.978325    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/45cbefc2-d0c1-41ac-a78e-be14100134c4-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-9dvjl\" (UID: \"45cbefc2-d0c1-41ac-a78e-be14100134c4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-9dvjl"
Jul 17 12:31:33.978348 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.978349    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b9cd22df-55bb-4bfb-8108-8bbad86b57cd-node-exporter-wtmp\") pod \"node-exporter-6jrn7\" (UID: \"b9cd22df-55bb-4bfb-8108-8bbad86b57cd\") " pod="openshift-monitoring/node-exporter-6jrn7"
Jul 17 12:31:33.978629 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.978438    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9cd22df-55bb-4bfb-8108-8bbad86b57cd-metrics-client-ca\") pod \"node-exporter-6jrn7\" (UID: \"b9cd22df-55bb-4bfb-8108-8bbad86b57cd\") " pod="openshift-monitoring/node-exporter-6jrn7"
Jul 17 12:31:33.978629 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:33.978486    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45cbefc2-d0c1-41ac-a78e-be14100134c4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-9dvjl\" (UID: \"45cbefc2-d0c1-41ac-a78e-be14100134c4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-9dvjl"
Jul 17 12:31:34.079283 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.079247    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45cbefc2-d0c1-41ac-a78e-be14100134c4-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-9dvjl\" (UID: \"45cbefc2-d0c1-41ac-a78e-be14100134c4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-9dvjl"
Jul 17 12:31:34.079283 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.079288    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnfp5\" (UniqueName: \"kubernetes.io/projected/b9cd22df-55bb-4bfb-8108-8bbad86b57cd-kube-api-access-dnfp5\") pod \"node-exporter-6jrn7\" (UID: \"b9cd22df-55bb-4bfb-8108-8bbad86b57cd\") " pod="openshift-monitoring/node-exporter-6jrn7"
Jul 17 12:31:34.079505 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.079327    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9cd22df-55bb-4bfb-8108-8bbad86b57cd-sys\") pod \"node-exporter-6jrn7\" (UID: \"b9cd22df-55bb-4bfb-8108-8bbad86b57cd\") " pod="openshift-monitoring/node-exporter-6jrn7"
Jul 17 12:31:34.079505 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.079360    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b9cd22df-55bb-4bfb-8108-8bbad86b57cd-node-exporter-tls\") pod \"node-exporter-6jrn7\" (UID: \"b9cd22df-55bb-4bfb-8108-8bbad86b57cd\") " pod="openshift-monitoring/node-exporter-6jrn7"
Jul 17 12:31:34.079505 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.079384    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/45cbefc2-d0c1-41ac-a78e-be14100134c4-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-9dvjl\" (UID: \"45cbefc2-d0c1-41ac-a78e-be14100134c4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-9dvjl"
Jul 17 12:31:34.079505 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.079409    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b9cd22df-55bb-4bfb-8108-8bbad86b57cd-node-exporter-wtmp\") pod \"node-exporter-6jrn7\" (UID: \"b9cd22df-55bb-4bfb-8108-8bbad86b57cd\") " pod="openshift-monitoring/node-exporter-6jrn7"
Jul 17 12:31:34.079505 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.079442    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9cd22df-55bb-4bfb-8108-8bbad86b57cd-metrics-client-ca\") pod \"node-exporter-6jrn7\" (UID: \"b9cd22df-55bb-4bfb-8108-8bbad86b57cd\") " pod="openshift-monitoring/node-exporter-6jrn7"
Jul 17 12:31:34.079505 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.079467    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9cd22df-55bb-4bfb-8108-8bbad86b57cd-sys\") pod \"node-exporter-6jrn7\" (UID: \"b9cd22df-55bb-4bfb-8108-8bbad86b57cd\") " pod="openshift-monitoring/node-exporter-6jrn7"
Jul 17 12:31:34.079805 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:34.079510    2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 17 12:31:34.079805 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:31:34.079597    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9cd22df-55bb-4bfb-8108-8bbad86b57cd-node-exporter-tls podName:b9cd22df-55bb-4bfb-8108-8bbad86b57cd nodeName:}" failed. No retries permitted until 2026-07-17 12:31:34.579575871 +0000 UTC m=+104.159687871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b9cd22df-55bb-4bfb-8108-8bbad86b57cd-node-exporter-tls") pod "node-exporter-6jrn7" (UID: "b9cd22df-55bb-4bfb-8108-8bbad86b57cd") : secret "node-exporter-tls" not found
Jul 17 12:31:34.079805 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.079621    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b9cd22df-55bb-4bfb-8108-8bbad86b57cd-node-exporter-wtmp\") pod \"node-exporter-6jrn7\" (UID: \"b9cd22df-55bb-4bfb-8108-8bbad86b57cd\") " pod="openshift-monitoring/node-exporter-6jrn7"
Jul 17 12:31:34.079805 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.079657    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45cbefc2-d0c1-41ac-a78e-be14100134c4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-9dvjl\" (UID: \"45cbefc2-d0c1-41ac-a78e-be14100134c4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-9dvjl"
Jul 17 12:31:34.079805 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.079713    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b9cd22df-55bb-4bfb-8108-8bbad86b57cd-root\") pod \"node-exporter-6jrn7\" (UID: \"b9cd22df-55bb-4bfb-8108-8bbad86b57cd\") " pod="openshift-monitoring/node-exporter-6jrn7"
Jul 17 12:31:34.079805 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.079755    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/45cbefc2-d0c1-41ac-a78e-be14100134c4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-9dvjl\" (UID: \"45cbefc2-d0c1-41ac-a78e-be14100134c4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-9dvjl"
Jul 17 12:31:34.079805 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.079785    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-644fv\" (UniqueName: \"kubernetes.io/projected/45cbefc2-d0c1-41ac-a78e-be14100134c4-kube-api-access-644fv\") pod \"kube-state-metrics-56c5594674-9dvjl\" (UID: \"45cbefc2-d0c1-41ac-a78e-be14100134c4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-9dvjl"
Jul 17 12:31:34.080139 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.079878    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b9cd22df-55bb-4bfb-8108-8bbad86b57cd-root\") pod \"node-exporter-6jrn7\" (UID: \"b9cd22df-55bb-4bfb-8108-8bbad86b57cd\") " pod="openshift-monitoring/node-exporter-6jrn7"
Jul 17 12:31:34.080139 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.079928    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b9cd22df-55bb-4bfb-8108-8bbad86b57cd-node-exporter-textfile\") pod \"node-exporter-6jrn7\" (UID: \"b9cd22df-55bb-4bfb-8108-8bbad86b57cd\") " pod="openshift-monitoring/node-exporter-6jrn7"
Jul 17 12:31:34.080139 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.079965    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b9cd22df-55bb-4bfb-8108-8bbad86b57cd-node-exporter-accelerators-collector-config\") pod \"node-exporter-6jrn7\" (UID: \"b9cd22df-55bb-4bfb-8108-8bbad86b57cd\") " pod="openshift-monitoring/node-exporter-6jrn7"
Jul 17 12:31:34.080139 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.079993    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/45cbefc2-d0c1-41ac-a78e-be14100134c4-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-9dvjl\" (UID: \"45cbefc2-d0c1-41ac-a78e-be14100134c4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-9dvjl"
Jul 17 12:31:34.080139 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.080004    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9cd22df-55bb-4bfb-8108-8bbad86b57cd-metrics-client-ca\") pod \"node-exporter-6jrn7\" (UID: \"b9cd22df-55bb-4bfb-8108-8bbad86b57cd\") " pod="openshift-monitoring/node-exporter-6jrn7"
Jul 17 12:31:34.080139 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.080036    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9cd22df-55bb-4bfb-8108-8bbad86b57cd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6jrn7\" (UID: \"b9cd22df-55bb-4bfb-8108-8bbad86b57cd\") " pod="openshift-monitoring/node-exporter-6jrn7"
Jul 17 12:31:34.080139 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.080093    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45cbefc2-d0c1-41ac-a78e-be14100134c4-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-9dvjl\" (UID: \"45cbefc2-d0c1-41ac-a78e-be14100134c4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-9dvjl"
Jul 17 12:31:34.080465 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.080423    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/45cbefc2-d0c1-41ac-a78e-be14100134c4-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-9dvjl\" (UID: \"45cbefc2-d0c1-41ac-a78e-be14100134c4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-9dvjl"
Jul 17 12:31:34.080549 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.080522    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b9cd22df-55bb-4bfb-8108-8bbad86b57cd-node-exporter-accelerators-collector-config\") pod \"node-exporter-6jrn7\" (UID: \"b9cd22df-55bb-4bfb-8108-8bbad86b57cd\") " pod="openshift-monitoring/node-exporter-6jrn7"
Jul 17 12:31:34.080690 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.080558    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/45cbefc2-d0c1-41ac-a78e-be14100134c4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-9dvjl\" (UID: \"45cbefc2-d0c1-41ac-a78e-be14100134c4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-9dvjl"
Jul 17 12:31:34.080781 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.080753    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b9cd22df-55bb-4bfb-8108-8bbad86b57cd-node-exporter-textfile\") pod \"node-exporter-6jrn7\" (UID: \"b9cd22df-55bb-4bfb-8108-8bbad86b57cd\") " pod="openshift-monitoring/node-exporter-6jrn7"
Jul 17 12:31:34.082331 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.082307    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9cd22df-55bb-4bfb-8108-8bbad86b57cd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6jrn7\" (UID: \"b9cd22df-55bb-4bfb-8108-8bbad86b57cd\") " pod="openshift-monitoring/node-exporter-6jrn7"
Jul 17 12:31:34.082657 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.082628    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45cbefc2-d0c1-41ac-a78e-be14100134c4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-9dvjl\" (UID: \"45cbefc2-d0c1-41ac-a78e-be14100134c4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-9dvjl"
Jul 17 12:31:34.082775 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.082679    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/45cbefc2-d0c1-41ac-a78e-be14100134c4-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-9dvjl\" (UID: \"45cbefc2-d0c1-41ac-a78e-be14100134c4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-9dvjl"
Jul 17 12:31:34.090352 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.090299    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnfp5\" (UniqueName: \"kubernetes.io/projected/b9cd22df-55bb-4bfb-8108-8bbad86b57cd-kube-api-access-dnfp5\") pod \"node-exporter-6jrn7\" (UID: \"b9cd22df-55bb-4bfb-8108-8bbad86b57cd\") " pod="openshift-monitoring/node-exporter-6jrn7"
Jul 17 12:31:34.090494 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.090477    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-644fv\" (UniqueName: \"kubernetes.io/projected/45cbefc2-d0c1-41ac-a78e-be14100134c4-kube-api-access-644fv\") pod \"kube-state-metrics-56c5594674-9dvjl\" (UID: \"45cbefc2-d0c1-41ac-a78e-be14100134c4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-9dvjl"
Jul 17 12:31:34.164870 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.164834    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-9dvjl"
Jul 17 12:31:34.322837 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.322813    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-9dvjl"]
Jul 17 12:31:34.324591 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:31:34.324566    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45cbefc2_d0c1_41ac_a78e_be14100134c4.slice/crio-28dbe8f066e4915f44d6974c45076ef2fc2e6490f2cc1ac5a449f0f997a2d836 WatchSource:0}: Error finding container 28dbe8f066e4915f44d6974c45076ef2fc2e6490f2cc1ac5a449f0f997a2d836: Status 404 returned error can't find the container with id 28dbe8f066e4915f44d6974c45076ef2fc2e6490f2cc1ac5a449f0f997a2d836
Jul 17 12:31:34.550587 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.550551    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-9dvjl" event={"ID":"45cbefc2-d0c1-41ac-a78e-be14100134c4","Type":"ContainerStarted","Data":"28dbe8f066e4915f44d6974c45076ef2fc2e6490f2cc1ac5a449f0f997a2d836"}
Jul 17 12:31:34.552211 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.552186    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vd7js" event={"ID":"9fe27e83-0d04-4f44-9b37-f410e08e5f51","Type":"ContainerStarted","Data":"26124cc14f1b6fb0f2f16a5b2874b042bc0d8b41c1f257152e2945e1bcd45f9b"}
Jul 17 12:31:34.584659 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.584625    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b9cd22df-55bb-4bfb-8108-8bbad86b57cd-node-exporter-tls\") pod \"node-exporter-6jrn7\" (UID: \"b9cd22df-55bb-4bfb-8108-8bbad86b57cd\") " pod="openshift-monitoring/node-exporter-6jrn7"
Jul 17 12:31:34.586923 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.586895    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b9cd22df-55bb-4bfb-8108-8bbad86b57cd-node-exporter-tls\") pod \"node-exporter-6jrn7\" (UID: \"b9cd22df-55bb-4bfb-8108-8bbad86b57cd\") " pod="openshift-monitoring/node-exporter-6jrn7"
Jul 17 12:31:34.772840 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:34.772806    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6jrn7"
Jul 17 12:31:34.783197 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:31:34.783165    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9cd22df_55bb_4bfb_8108_8bbad86b57cd.slice/crio-d6827c00740cc6f754dbbd3d5523a90a9c8476e88a1fe93be95bd6375875af03 WatchSource:0}: Error finding container d6827c00740cc6f754dbbd3d5523a90a9c8476e88a1fe93be95bd6375875af03: Status 404 returned error can't find the container with id d6827c00740cc6f754dbbd3d5523a90a9c8476e88a1fe93be95bd6375875af03
Jul 17 12:31:35.290181 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:35.290122    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/efb8a146-b5cf-4b54-bb64-3b060a736ca4-metrics-tls\") pod \"dns-default-wkj54\" (UID: \"efb8a146-b5cf-4b54-bb64-3b060a736ca4\") " pod="openshift-dns/dns-default-wkj54"
Jul 17 12:31:35.293082 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:35.293055    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/efb8a146-b5cf-4b54-bb64-3b060a736ca4-metrics-tls\") pod \"dns-default-wkj54\" (UID: \"efb8a146-b5cf-4b54-bb64-3b060a736ca4\") " pod="openshift-dns/dns-default-wkj54"
Jul 17 12:31:35.542943 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:35.542908    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bjllw\""
Jul 17 12:31:35.551624 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:35.551343    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wkj54"
Jul 17 12:31:35.558613 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:35.558576    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6jrn7" event={"ID":"b9cd22df-55bb-4bfb-8108-8bbad86b57cd","Type":"ContainerStarted","Data":"d6827c00740cc6f754dbbd3d5523a90a9c8476e88a1fe93be95bd6375875af03"}
Jul 17 12:31:36.183863 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:36.183643    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wkj54"]
Jul 17 12:31:36.565604 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:36.565558    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-9dvjl" event={"ID":"45cbefc2-d0c1-41ac-a78e-be14100134c4","Type":"ContainerStarted","Data":"e90e814437e88e1427275e67df3149261bd36e02863aa7e3c5fcef0a29eeda8f"}
Jul 17 12:31:36.566096 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:36.565643    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-9dvjl" event={"ID":"45cbefc2-d0c1-41ac-a78e-be14100134c4","Type":"ContainerStarted","Data":"97b1ceceafdc84cef582ee3fecf84b154171779eea24957739c068e07cc22f8a"}
Jul 17 12:31:36.566096 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:36.565669    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-9dvjl" event={"ID":"45cbefc2-d0c1-41ac-a78e-be14100134c4","Type":"ContainerStarted","Data":"4da93ee332fcd56e60940d4fd8fbe267d84fdb0e4ce96c784cbfa4f2d703263b"}
Jul 17 12:31:36.567488 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:36.567460    2564 generic.go:356] "Generic (PLEG): container finished" podID="b9cd22df-55bb-4bfb-8108-8bbad86b57cd" containerID="820aca5f00a3f223e5cd01f83f86e3f55e159393653c7ed0b60a3d63e2135fdc" exitCode=0
Jul 17 12:31:36.567615 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:36.567552    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6jrn7" event={"ID":"b9cd22df-55bb-4bfb-8108-8bbad86b57cd","Type":"ContainerDied","Data":"820aca5f00a3f223e5cd01f83f86e3f55e159393653c7ed0b60a3d63e2135fdc"}
Jul 17 12:31:36.569311 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:36.569288    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wkj54" event={"ID":"efb8a146-b5cf-4b54-bb64-3b060a736ca4","Type":"ContainerStarted","Data":"a341058d0ef975014cfa4393139ccaf7aa0778401fef8dd0bcc2755467db37f1"}
Jul 17 12:31:36.586066 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:36.585999    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-9dvjl" podStartSLOduration=1.904265533 podStartE2EDuration="3.585979842s" totalImagesPullingTime="1.681714309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:31:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:31:36.584844961 +0000 UTC m=+106.164956996" watchObservedRunningTime="2026-07-17 12:31:36.585979842 +0000 UTC m=+106.166091861"
Jul 17 12:31:38.163607 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:38.163571    2564 patch_prober.go:28] interesting pod/image-registry-79777dbb55-7fclv container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 17 12:31:38.163996 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:38.163630    2564 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-79777dbb55-7fclv" podUID="4e4764c4-743b-4010-a3f1-57b54e8b80a5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:31:38.580047 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:38.580006    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6jrn7" event={"ID":"b9cd22df-55bb-4bfb-8108-8bbad86b57cd","Type":"ContainerStarted","Data":"df44dbd32218c1f7b5547a52f45aacff45a144aff827ed4ab1454d45037da8c0"}
Jul 17 12:31:38.580212 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:38.580071    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6jrn7" event={"ID":"b9cd22df-55bb-4bfb-8108-8bbad86b57cd","Type":"ContainerStarted","Data":"c60a0f6658edf64291ac9ba37409e4fd4e64bd208404b9a76d39d96c833f8843"}
Jul 17 12:31:38.582480 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:38.582448    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vd7js" event={"ID":"9fe27e83-0d04-4f44-9b37-f410e08e5f51","Type":"ContainerStarted","Data":"7b3a140b0796b68669b28f0763f22729c9caffe3b3713c0bfbfd073e36016a28"}
Jul 17 12:31:38.601672 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:38.601608    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6jrn7" podStartSLOduration=4.265630805 podStartE2EDuration="5.601585696s" totalImagesPullingTime="1.225201771s" totalInitContainerRuntime="110.75312ms" isStatefulPod=true podCreationTimestamp="2026-07-17 12:31:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:31:38.599448818 +0000 UTC m=+108.179560871" watchObservedRunningTime="2026-07-17 12:31:38.601585696 +0000 UTC m=+108.181697717"
Jul 17 12:31:38.618978 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:38.618918    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vd7js" podStartSLOduration=16.607620741 podStartE2EDuration="29.618901044s" totalImagesPullingTime="13.011280303s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:31:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:31:38.617549859 +0000 UTC m=+108.197661880" watchObservedRunningTime="2026-07-17 12:31:38.618901044 +0000 UTC m=+108.199013062"
Jul 17 12:31:39.026366 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.026280    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-57bf6df894-8xmnn"]
Jul 17 12:31:39.031406 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.031381    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn"
Jul 17 12:31:39.034678 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.034627    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-zqmzt\""
Jul 17 12:31:39.035308 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.035277    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 17 12:31:39.035464 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.035446    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 17 12:31:39.035956 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.035938    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 17 12:31:39.036281 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.036248    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 17 12:31:39.036281 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.036277    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 17 12:31:39.042957 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.042927    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 17 12:31:39.055206 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.055169    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-57bf6df894-8xmnn"]
Jul 17 12:31:39.229644 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.229516    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e457f9d0-1192-4bfe-a11d-5bad8bc2c138-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-57bf6df894-8xmnn\" (UID: \"e457f9d0-1192-4bfe-a11d-5bad8bc2c138\") " pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn"
Jul 17 12:31:39.229644 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.229572    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e457f9d0-1192-4bfe-a11d-5bad8bc2c138-telemeter-trusted-ca-bundle\") pod \"telemeter-client-57bf6df894-8xmnn\" (UID: \"e457f9d0-1192-4bfe-a11d-5bad8bc2c138\") " pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn"
Jul 17 12:31:39.229644 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.229614    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbqrr\" (UniqueName: \"kubernetes.io/projected/e457f9d0-1192-4bfe-a11d-5bad8bc2c138-kube-api-access-lbqrr\") pod \"telemeter-client-57bf6df894-8xmnn\" (UID: \"e457f9d0-1192-4bfe-a11d-5bad8bc2c138\") " pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn"
Jul 17 12:31:39.230293 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.229670    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e457f9d0-1192-4bfe-a11d-5bad8bc2c138-telemeter-client-tls\") pod \"telemeter-client-57bf6df894-8xmnn\" (UID: \"e457f9d0-1192-4bfe-a11d-5bad8bc2c138\") " pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn"
Jul 17 12:31:39.230293 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.229730    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e457f9d0-1192-4bfe-a11d-5bad8bc2c138-secret-telemeter-client\") pod \"telemeter-client-57bf6df894-8xmnn\" (UID: \"e457f9d0-1192-4bfe-a11d-5bad8bc2c138\") " pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn"
Jul 17 12:31:39.230293 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.229765    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e457f9d0-1192-4bfe-a11d-5bad8bc2c138-federate-client-tls\") pod \"telemeter-client-57bf6df894-8xmnn\" (UID: \"e457f9d0-1192-4bfe-a11d-5bad8bc2c138\") " pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn"
Jul 17 12:31:39.230293 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.229879    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e457f9d0-1192-4bfe-a11d-5bad8bc2c138-serving-certs-ca-bundle\") pod \"telemeter-client-57bf6df894-8xmnn\" (UID: \"e457f9d0-1192-4bfe-a11d-5bad8bc2c138\") " pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn"
Jul 17 12:31:39.230293 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.229918    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e457f9d0-1192-4bfe-a11d-5bad8bc2c138-metrics-client-ca\") pod \"telemeter-client-57bf6df894-8xmnn\" (UID: \"e457f9d0-1192-4bfe-a11d-5bad8bc2c138\") " pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn"
Jul 17 12:31:39.331090 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.331048    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e457f9d0-1192-4bfe-a11d-5bad8bc2c138-telemeter-client-tls\") pod \"telemeter-client-57bf6df894-8xmnn\" (UID: \"e457f9d0-1192-4bfe-a11d-5bad8bc2c138\") " pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn"
Jul 17 12:31:39.331264 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.331140    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e457f9d0-1192-4bfe-a11d-5bad8bc2c138-secret-telemeter-client\") pod \"telemeter-client-57bf6df894-8xmnn\" (UID: \"e457f9d0-1192-4bfe-a11d-5bad8bc2c138\") " pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn"
Jul 17 12:31:39.331264 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.331205    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e457f9d0-1192-4bfe-a11d-5bad8bc2c138-federate-client-tls\") pod \"telemeter-client-57bf6df894-8xmnn\" (UID: \"e457f9d0-1192-4bfe-a11d-5bad8bc2c138\") " pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn"
Jul 17 12:31:39.331396 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.331265    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e457f9d0-1192-4bfe-a11d-5bad8bc2c138-serving-certs-ca-bundle\") pod \"telemeter-client-57bf6df894-8xmnn\" (UID: \"e457f9d0-1192-4bfe-a11d-5bad8bc2c138\") " pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn"
Jul 17 12:31:39.331396 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.331319    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e457f9d0-1192-4bfe-a11d-5bad8bc2c138-metrics-client-ca\") pod \"telemeter-client-57bf6df894-8xmnn\" (UID: \"e457f9d0-1192-4bfe-a11d-5bad8bc2c138\") " pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn"
Jul 17 12:31:39.331396 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.331353    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e457f9d0-1192-4bfe-a11d-5bad8bc2c138-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-57bf6df894-8xmnn\" (UID: \"e457f9d0-1192-4bfe-a11d-5bad8bc2c138\") " pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn"
Jul 17 12:31:39.331396 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.331382    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e457f9d0-1192-4bfe-a11d-5bad8bc2c138-telemeter-trusted-ca-bundle\") pod \"telemeter-client-57bf6df894-8xmnn\" (UID: \"e457f9d0-1192-4bfe-a11d-5bad8bc2c138\") " pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn"
Jul 17 12:31:39.331650 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.331412    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbqrr\" (UniqueName: \"kubernetes.io/projected/e457f9d0-1192-4bfe-a11d-5bad8bc2c138-kube-api-access-lbqrr\") pod \"telemeter-client-57bf6df894-8xmnn\" (UID: \"e457f9d0-1192-4bfe-a11d-5bad8bc2c138\") " pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn"
Jul 17 12:31:39.332377 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.332346    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e457f9d0-1192-4bfe-a11d-5bad8bc2c138-metrics-client-ca\") pod \"telemeter-client-57bf6df894-8xmnn\" (UID: \"e457f9d0-1192-4bfe-a11d-5bad8bc2c138\") " pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn"
Jul 17 12:31:39.333978 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.332954    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e457f9d0-1192-4bfe-a11d-5bad8bc2c138-serving-certs-ca-bundle\") pod \"telemeter-client-57bf6df894-8xmnn\" (UID: \"e457f9d0-1192-4bfe-a11d-5bad8bc2c138\") " pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn"
Jul 17 12:31:39.333978 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.333749    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e457f9d0-1192-4bfe-a11d-5bad8bc2c138-telemeter-trusted-ca-bundle\") pod \"telemeter-client-57bf6df894-8xmnn\" (UID: \"e457f9d0-1192-4bfe-a11d-5bad8bc2c138\") " pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn"
Jul 17 12:31:39.334292 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.334250    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e457f9d0-1192-4bfe-a11d-5bad8bc2c138-secret-telemeter-client\") pod \"telemeter-client-57bf6df894-8xmnn\" (UID: \"e457f9d0-1192-4bfe-a11d-5bad8bc2c138\") " pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn"
Jul 17 12:31:39.334392 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.334345    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e457f9d0-1192-4bfe-a11d-5bad8bc2c138-federate-client-tls\") pod \"telemeter-client-57bf6df894-8xmnn\" (UID: \"e457f9d0-1192-4bfe-a11d-5bad8bc2c138\") " pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn"
Jul 17 12:31:39.334595 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.334554    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e457f9d0-1192-4bfe-a11d-5bad8bc2c138-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-57bf6df894-8xmnn\" (UID: \"e457f9d0-1192-4bfe-a11d-5bad8bc2c138\") " pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn"
Jul 17 12:31:39.334724 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.334682    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e457f9d0-1192-4bfe-a11d-5bad8bc2c138-telemeter-client-tls\") pod \"telemeter-client-57bf6df894-8xmnn\" (UID: \"e457f9d0-1192-4bfe-a11d-5bad8bc2c138\") " pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn"
Jul 17 12:31:39.345749 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.345680    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbqrr\" (UniqueName: \"kubernetes.io/projected/e457f9d0-1192-4bfe-a11d-5bad8bc2c138-kube-api-access-lbqrr\") pod \"telemeter-client-57bf6df894-8xmnn\" (UID: \"e457f9d0-1192-4bfe-a11d-5bad8bc2c138\") " pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn"
Jul 17 12:31:39.643866 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.643777    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn"
Jul 17 12:31:39.959856 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:39.959824    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-57bf6df894-8xmnn"]
Jul 17 12:31:39.961358 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:31:39.961327    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode457f9d0_1192_4bfe_a11d_5bad8bc2c138.slice/crio-7494f1698f3a1b77db06d1ff8cc6fa5c87116720fcb1b9cbe9e3b38234efe4bb WatchSource:0}: Error finding container 7494f1698f3a1b77db06d1ff8cc6fa5c87116720fcb1b9cbe9e3b38234efe4bb: Status 404 returned error can't find the container with id 7494f1698f3a1b77db06d1ff8cc6fa5c87116720fcb1b9cbe9e3b38234efe4bb
Jul 17 12:31:40.011108 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.011074    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 12:31:40.033639 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.033616    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.037621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.036673    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 17 12:31:40.037621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.036810    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 17 12:31:40.037621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.036676    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 17 12:31:40.037621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.037131    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1km0rejukgsc8\""
Jul 17 12:31:40.037621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.037365    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 17 12:31:40.037621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.037617    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 17 12:31:40.038813 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.038462    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 17 12:31:40.038813 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.038465    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 17 12:31:40.038813 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.038499    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 17 12:31:40.038813 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.038528    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 17 12:31:40.038813 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.038646    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 17 12:31:40.039606 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.039176    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 17 12:31:40.039606 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.039400    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 17 12:31:40.039606 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.039604    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-g2sbh\""
Jul 17 12:31:40.040564 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.040259    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 17 12:31:40.043007 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.042973    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 17 12:31:40.043755 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.043731    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 12:31:40.045018 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.044999    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 17 12:31:40.137949 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.137926    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.138048 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.137970    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/174db30d-e76a-49c6-9496-b61e83a4cfae-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.138048 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.137989    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/174db30d-e76a-49c6-9496-b61e83a4cfae-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.138048 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.138004    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.138048 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.138021    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfgpp\" (UniqueName: \"kubernetes.io/projected/174db30d-e76a-49c6-9496-b61e83a4cfae-kube-api-access-qfgpp\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.138193 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.138082    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.138193 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.138128    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.138193 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.138157    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-web-config\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.138298 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.138192    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.138298 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.138222    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.138365 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.138295    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.138365 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.138330    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.138440 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.138383    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.138440 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.138430    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/174db30d-e76a-49c6-9496-b61e83a4cfae-config-out\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.138511 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.138497    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.138564 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.138546    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.138690 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.138670    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.138765 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.138721    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.138765 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.138754    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.138836 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.138769    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-config\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.240068 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.239964    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.240068 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.240010    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.240068 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.240041    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.240656 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.240069    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/174db30d-e76a-49c6-9496-b61e83a4cfae-config-out\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.240656 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.240252    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.240656 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.240287    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.240656 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.240323    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.240656 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.240353    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.240656 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.240386    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.240656 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.240414    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-config\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.240656 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.240449    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.240656 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.240482    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/174db30d-e76a-49c6-9496-b61e83a4cfae-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.240656 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.240507    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/174db30d-e76a-49c6-9496-b61e83a4cfae-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.240656 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.240543    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.240656 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.240581    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfgpp\" (UniqueName: \"kubernetes.io/projected/174db30d-e76a-49c6-9496-b61e83a4cfae-kube-api-access-qfgpp\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.240656 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.240607    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.240656 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.240633    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.240656 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.240652    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-web-config\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.241479 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.240674    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.241479 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.240719    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.241479 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.240929    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.241479 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.241139    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.241479 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.241324    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.243181 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.242112    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.243181 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.242414    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/174db30d-e76a-49c6-9496-b61e83a4cfae-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.243181 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.243126    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.244247 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.243615    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.244247 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.243669    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/174db30d-e76a-49c6-9496-b61e83a4cfae-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.244247 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.244101    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.244247 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.244179    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/174db30d-e76a-49c6-9496-b61e83a4cfae-config-out\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.244516 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.244317    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.244573 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.244513    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.244774 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.244751    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-web-config\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.245618 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.245582    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-config\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.245743 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.245725    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.245816 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.245761    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.245816 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.245776    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.245885 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.245818    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.246442 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.246424    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.252315 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.252292    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfgpp\" (UniqueName: \"kubernetes.io/projected/174db30d-e76a-49c6-9496-b61e83a4cfae-kube-api-access-qfgpp\") pod \"prometheus-k8s-0\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.348566 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.348526    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:40.487746 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.487718    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 12:31:40.489373 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:31:40.489344    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod174db30d_e76a_49c6_9496_b61e83a4cfae.slice/crio-9fba51fae4ab29a1eea473ca9543ec1d80dabca4d3e115b328c37417e20c754d WatchSource:0}: Error finding container 9fba51fae4ab29a1eea473ca9543ec1d80dabca4d3e115b328c37417e20c754d: Status 404 returned error can't find the container with id 9fba51fae4ab29a1eea473ca9543ec1d80dabca4d3e115b328c37417e20c754d
Jul 17 12:31:40.501543 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.501500    2564 patch_prober.go:28] interesting pod/image-registry-79777dbb55-7fclv container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 17 12:31:40.501659 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.501563    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-79777dbb55-7fclv" podUID="4e4764c4-743b-4010-a3f1-57b54e8b80a5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:31:40.597347 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.597309    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wkj54" event={"ID":"efb8a146-b5cf-4b54-bb64-3b060a736ca4","Type":"ContainerStarted","Data":"fae73feced30f6e11ad2ecdcc3bdf78a4e2d7ea735380a21d0b3757b90ef2c78"}
Jul 17 12:31:40.597673 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.597401    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wkj54" event={"ID":"efb8a146-b5cf-4b54-bb64-3b060a736ca4","Type":"ContainerStarted","Data":"37f021b6997e10a8e30a4c6313a57afee46de28567c3817d013affd413b8e68d"}
Jul 17 12:31:40.597673 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.597437    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wkj54"
Jul 17 12:31:40.598790 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.598762    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"174db30d-e76a-49c6-9496-b61e83a4cfae","Type":"ContainerStarted","Data":"9fba51fae4ab29a1eea473ca9543ec1d80dabca4d3e115b328c37417e20c754d"}
Jul 17 12:31:40.600131 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.600101    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn" event={"ID":"e457f9d0-1192-4bfe-a11d-5bad8bc2c138","Type":"ContainerStarted","Data":"7494f1698f3a1b77db06d1ff8cc6fa5c87116720fcb1b9cbe9e3b38234efe4bb"}
Jul 17 12:31:40.618449 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:40.618392    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wkj54" podStartSLOduration=65.989174616 podStartE2EDuration="1m9.618371761s" totalImagesPullingTime="3.629197145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:30:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:31:40.617781224 +0000 UTC m=+110.197893244" watchObservedRunningTime="2026-07-17 12:31:40.618371761 +0000 UTC m=+110.198483784"
Jul 17 12:31:42.612148 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:42.612116    2564 generic.go:356] "Generic (PLEG): container finished" podID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerID="b3e402c723dd83efeeaa197dc0b97f8237d35d6c87a5bd106e0390c11e798f9c" exitCode=0
Jul 17 12:31:42.612739 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:42.612190    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"174db30d-e76a-49c6-9496-b61e83a4cfae","Type":"ContainerDied","Data":"b3e402c723dd83efeeaa197dc0b97f8237d35d6c87a5bd106e0390c11e798f9c"}
Jul 17 12:31:42.614422 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:42.614389    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn" event={"ID":"e457f9d0-1192-4bfe-a11d-5bad8bc2c138","Type":"ContainerStarted","Data":"67ac8218d6a47a677ca0e7f29dd15a1bb09d6f6ae23be238e1948f26ce7a8409"}
Jul 17 12:31:42.614512 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:42.614447    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn" event={"ID":"e457f9d0-1192-4bfe-a11d-5bad8bc2c138","Type":"ContainerStarted","Data":"4c402d2c59a9efb6ad3585f3b0c86b96077c9449c91f886ae133a13446966c57"}
Jul 17 12:31:42.614512 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:42.614468    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn" event={"ID":"e457f9d0-1192-4bfe-a11d-5bad8bc2c138","Type":"ContainerStarted","Data":"75834cc48f8f86231c9f46026b204152521af25ec11f3b361a56157226b87216"}
Jul 17 12:31:42.671173 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:42.671118    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-57bf6df894-8xmnn" podStartSLOduration=1.678580583 podStartE2EDuration="3.671100979s" totalImagesPullingTime="1.992520396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:31:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:31:42.670209982 +0000 UTC m=+112.250322000" watchObservedRunningTime="2026-07-17 12:31:42.671100979 +0000 UTC m=+112.251212998"
Jul 17 12:31:45.628794 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:45.628748    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"174db30d-e76a-49c6-9496-b61e83a4cfae","Type":"ContainerStarted","Data":"d7b6d132a0bb7594e53a09adaeba3b7ebba4cd2f78d383d4cc97c71f899484cf"}
Jul 17 12:31:46.633865 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:46.633826    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"174db30d-e76a-49c6-9496-b61e83a4cfae","Type":"ContainerStarted","Data":"9c3bf91046735e367f7ea162216e9c871f1d00053826ad385e2c1d812a32a8c7"}
Jul 17 12:31:47.641217 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:47.641184    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"174db30d-e76a-49c6-9496-b61e83a4cfae","Type":"ContainerStarted","Data":"b20bd13119c600fa40045b72adde2bdd6ac2c20bc854b571ba8239bb095d7c0b"}
Jul 17 12:31:47.641584 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:47.641269    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"174db30d-e76a-49c6-9496-b61e83a4cfae","Type":"ContainerStarted","Data":"1ea58e02b641e26f9f343c7d75eea454bb6a28a3208285fb24b75b5fe3641b7e"}
Jul 17 12:31:47.641584 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:47.641314    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"174db30d-e76a-49c6-9496-b61e83a4cfae","Type":"ContainerStarted","Data":"032e09da137bf079916fd1e14e2ecb166d4f9526c6fab1ff1073e346c63f91f6"}
Jul 17 12:31:48.163040 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:48.163000    2564 patch_prober.go:28] interesting pod/image-registry-79777dbb55-7fclv container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 17 12:31:48.163284 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:48.163057    2564 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-79777dbb55-7fclv" podUID="4e4764c4-743b-4010-a3f1-57b54e8b80a5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:31:48.647795 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:48.647761    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"174db30d-e76a-49c6-9496-b61e83a4cfae","Type":"ContainerStarted","Data":"6fff370e6c1f236d5f1653db7c34f45da1bdd609b64fa0f3de2b4d8e570f9fcd"}
Jul 17 12:31:48.683516 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:48.683456    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.554707222 podStartE2EDuration="9.683439367s" totalImagesPullingTime="6.108976904s" totalInitContainerRuntime="19.755241ms" isStatefulPod=false podCreationTimestamp="2026-07-17 12:31:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:31:48.683034406 +0000 UTC m=+118.263146450" watchObservedRunningTime="2026-07-17 12:31:48.683439367 +0000 UTC m=+118.263551385"
Jul 17 12:31:50.349553 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:50.349494    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:31:50.501639 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:50.501599    2564 patch_prober.go:28] interesting pod/image-registry-79777dbb55-7fclv container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 17 12:31:50.501807 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:50.501662    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-79777dbb55-7fclv" podUID="4e4764c4-743b-4010-a3f1-57b54e8b80a5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:31:50.609087 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:50.609011    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wkj54"
Jul 17 12:31:55.013507 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:55.013467    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:31:58.162667 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:58.162632    2564 patch_prober.go:28] interesting pod/image-registry-79777dbb55-7fclv container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 17 12:31:58.163105 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:58.162724    2564 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-79777dbb55-7fclv" podUID="4e4764c4-743b-4010-a3f1-57b54e8b80a5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:31:58.163105 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:58.162774    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:31:58.163326 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:58.163293    2564 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="registry" containerStatusID={"Type":"cri-o","ID":"ef3c43920f78334dbf5258c279918b7d79863834d9632d51480164bc2c57b924"} pod="openshift-image-registry/image-registry-79777dbb55-7fclv" containerMessage="Container registry failed liveness probe, will be restarted"
Jul 17 12:31:58.166948 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:58.166925    2564 patch_prober.go:28] interesting pod/image-registry-79777dbb55-7fclv container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 17 12:31:58.167062 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:31:58.166977    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-79777dbb55-7fclv" podUID="4e4764c4-743b-4010-a3f1-57b54e8b80a5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:32:00.733373 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:00.733318    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/937e7356-528f-4a56-a80e-9486657ec301-metrics-certs\") pod \"network-metrics-daemon-lxrxw\" (UID: \"937e7356-528f-4a56-a80e-9486657ec301\") " pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:32:00.735719 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:00.735683    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/937e7356-528f-4a56-a80e-9486657ec301-metrics-certs\") pod \"network-metrics-daemon-lxrxw\" (UID: \"937e7356-528f-4a56-a80e-9486657ec301\") " pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:32:00.923218 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:00.923188    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wrsj9\""
Jul 17 12:32:00.930289 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:00.930262    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lxrxw"
Jul 17 12:32:01.072980 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:01.072943    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lxrxw"]
Jul 17 12:32:01.076433 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:32:01.076356    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod937e7356_528f_4a56_a80e_9486657ec301.slice/crio-27286f85d598c9756dedb1f29e4334c1ab07f6cdc27d288b99e4e490a94c71f9 WatchSource:0}: Error finding container 27286f85d598c9756dedb1f29e4334c1ab07f6cdc27d288b99e4e490a94c71f9: Status 404 returned error can't find the container with id 27286f85d598c9756dedb1f29e4334c1ab07f6cdc27d288b99e4e490a94c71f9
Jul 17 12:32:01.695526 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:01.695464    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lxrxw" event={"ID":"937e7356-528f-4a56-a80e-9486657ec301","Type":"ContainerStarted","Data":"27286f85d598c9756dedb1f29e4334c1ab07f6cdc27d288b99e4e490a94c71f9"}
Jul 17 12:32:02.701135 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:02.701089    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lxrxw" event={"ID":"937e7356-528f-4a56-a80e-9486657ec301","Type":"ContainerStarted","Data":"39124c6cfe044217722286eb3c63a4098b1ade352df5c9dca348e22ddef99885"}
Jul 17 12:32:02.701513 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:02.701187    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lxrxw" event={"ID":"937e7356-528f-4a56-a80e-9486657ec301","Type":"ContainerStarted","Data":"e7248c56248737422e30c9f2125be86ac1c12031d0e33e02365434f57f419e04"}
Jul 17 12:32:02.719958 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:02.719898    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lxrxw" podStartSLOduration=130.413537729 podStartE2EDuration="2m11.719878486s" totalImagesPullingTime="1.306340757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:29:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:32:02.718861292 +0000 UTC m=+132.298973331" watchObservedRunningTime="2026-07-17 12:32:02.719878486 +0000 UTC m=+132.299990503"
Jul 17 12:32:08.167103 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:08.167074    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:32:09.525217 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:09.525176    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79777dbb55-7fclv"]
Jul 17 12:32:23.182810 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:23.182767    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-79777dbb55-7fclv" podUID="4e4764c4-743b-4010-a3f1-57b54e8b80a5" containerName="registry" containerID="cri-o://ef3c43920f78334dbf5258c279918b7d79863834d9632d51480164bc2c57b924" gracePeriod=30
Jul 17 12:32:24.785884 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:24.785845    2564 generic.go:356] "Generic (PLEG): container finished" podID="4e4764c4-743b-4010-a3f1-57b54e8b80a5" containerID="ef3c43920f78334dbf5258c279918b7d79863834d9632d51480164bc2c57b924" exitCode=0
Jul 17 12:32:24.786333 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:24.785920    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79777dbb55-7fclv" event={"ID":"4e4764c4-743b-4010-a3f1-57b54e8b80a5","Type":"ContainerDied","Data":"ef3c43920f78334dbf5258c279918b7d79863834d9632d51480164bc2c57b924"}
Jul 17 12:32:24.786333 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:24.785995    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79777dbb55-7fclv" event={"ID":"4e4764c4-743b-4010-a3f1-57b54e8b80a5","Type":"ContainerStarted","Data":"aa057a48328cd2d650cad06ba333dbd6e170c78bd5fd55abf033aa99db7b20b2"}
Jul 17 12:32:24.786333 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:24.786165    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:32:27.799148 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:27.799115    2564 generic.go:356] "Generic (PLEG): container finished" podID="a6b7914b-3758-4e34-bc34-309714b615c6" containerID="be335881e2ee0c3d42c5b77f7e38eeb98637b07277fd6451b33424f556cca350" exitCode=0
Jul 17 12:32:27.799503 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:27.799185    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-42w44" event={"ID":"a6b7914b-3758-4e34-bc34-309714b615c6","Type":"ContainerDied","Data":"be335881e2ee0c3d42c5b77f7e38eeb98637b07277fd6451b33424f556cca350"}
Jul 17 12:32:27.799503 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:27.799490    2564 scope.go:117] "RemoveContainer" containerID="be335881e2ee0c3d42c5b77f7e38eeb98637b07277fd6451b33424f556cca350"
Jul 17 12:32:28.806251 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:28.806205    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-42w44" event={"ID":"a6b7914b-3758-4e34-bc34-309714b615c6","Type":"ContainerStarted","Data":"1e1a6dec6c6b7ac1b8c567a4a0219e58285562bc3118a55efab31f5d33871436"}
Jul 17 12:32:38.847543 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:38.847506    2564 generic.go:356] "Generic (PLEG): container finished" podID="6652240d-b5e1-4079-b293-3e5d8469b4f9" containerID="37490c8825347400e70e6089267ec5244ae8c4807abd82722a9f13d1043ca0a6" exitCode=0
Jul 17 12:32:38.848054 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:38.847586    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-2pclj" event={"ID":"6652240d-b5e1-4079-b293-3e5d8469b4f9","Type":"ContainerDied","Data":"37490c8825347400e70e6089267ec5244ae8c4807abd82722a9f13d1043ca0a6"}
Jul 17 12:32:38.848054 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:38.848028    2564 scope.go:117] "RemoveContainer" containerID="37490c8825347400e70e6089267ec5244ae8c4807abd82722a9f13d1043ca0a6"
Jul 17 12:32:39.595886 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:39.595857    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-79777dbb55-7fclv_4e4764c4-743b-4010-a3f1-57b54e8b80a5/registry/0.log"
Jul 17 12:32:39.602387 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:39.602362    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-79777dbb55-7fclv_4e4764c4-743b-4010-a3f1-57b54e8b80a5/registry/1.log"
Jul 17 12:32:39.628167 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:39.628137    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xxld5_cc19f981-8855-47ce-98c7-6e8e9c0e65d2/node-ca/0.log"
Jul 17 12:32:39.853006 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:39.852918    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-2pclj" event={"ID":"6652240d-b5e1-4079-b293-3e5d8469b4f9","Type":"ContainerStarted","Data":"f2352af1b3a0fa28644ae06dd405f7a3231855569bfbd8a95c96b7e5fcc284eb"}
Jul 17 12:32:40.349076 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:40.349037    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:32:40.368108 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:40.368077    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:32:40.872752 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:40.872722    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:32:44.790856 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:44.790783    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:32:49.804968 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:49.804905    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-79777dbb55-7fclv" podUID="4e4764c4-743b-4010-a3f1-57b54e8b80a5" containerName="registry" containerID="cri-o://aa057a48328cd2d650cad06ba333dbd6e170c78bd5fd55abf033aa99db7b20b2" gracePeriod=30
Jul 17 12:32:50.059888 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.059821    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:32:50.147427 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.147390    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e4764c4-743b-4010-a3f1-57b54e8b80a5-image-registry-private-configuration\") pod \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") "
Jul 17 12:32:50.147427 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.147429    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e4764c4-743b-4010-a3f1-57b54e8b80a5-registry-certificates\") pod \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") "
Jul 17 12:32:50.147657 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.147530    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e4764c4-743b-4010-a3f1-57b54e8b80a5-ca-trust-extracted\") pod \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") "
Jul 17 12:32:50.147738 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.147677    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-729ww\" (UniqueName: \"kubernetes.io/projected/4e4764c4-743b-4010-a3f1-57b54e8b80a5-kube-api-access-729ww\") pod \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") "
Jul 17 12:32:50.147790 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.147769    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e4764c4-743b-4010-a3f1-57b54e8b80a5-installation-pull-secrets\") pod \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") "
Jul 17 12:32:50.147841 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.147807    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e4764c4-743b-4010-a3f1-57b54e8b80a5-registry-tls\") pod \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") "
Jul 17 12:32:50.147896 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.147852    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e4764c4-743b-4010-a3f1-57b54e8b80a5-bound-sa-token\") pod \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") "
Jul 17 12:32:50.147896 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.147887    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e4764c4-743b-4010-a3f1-57b54e8b80a5-trusted-ca\") pod \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\" (UID: \"4e4764c4-743b-4010-a3f1-57b54e8b80a5\") "
Jul 17 12:32:50.147994 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.147937    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4764c4-743b-4010-a3f1-57b54e8b80a5-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4e4764c4-743b-4010-a3f1-57b54e8b80a5" (UID: "4e4764c4-743b-4010-a3f1-57b54e8b80a5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:32:50.148259 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.148232    2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e4764c4-743b-4010-a3f1-57b54e8b80a5-registry-certificates\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:32:50.148403 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.148372    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4764c4-743b-4010-a3f1-57b54e8b80a5-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4e4764c4-743b-4010-a3f1-57b54e8b80a5" (UID: "4e4764c4-743b-4010-a3f1-57b54e8b80a5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:32:50.150370 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.150318    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4764c4-743b-4010-a3f1-57b54e8b80a5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4e4764c4-743b-4010-a3f1-57b54e8b80a5" (UID: "4e4764c4-743b-4010-a3f1-57b54e8b80a5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:32:50.150485 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.150450    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4764c4-743b-4010-a3f1-57b54e8b80a5-kube-api-access-729ww" (OuterVolumeSpecName: "kube-api-access-729ww") pod "4e4764c4-743b-4010-a3f1-57b54e8b80a5" (UID: "4e4764c4-743b-4010-a3f1-57b54e8b80a5"). InnerVolumeSpecName "kube-api-access-729ww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:32:50.151097 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.151061    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e4764c4-743b-4010-a3f1-57b54e8b80a5-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4e4764c4-743b-4010-a3f1-57b54e8b80a5" (UID: "4e4764c4-743b-4010-a3f1-57b54e8b80a5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:32:50.151487 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.151463    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e4764c4-743b-4010-a3f1-57b54e8b80a5-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4e4764c4-743b-4010-a3f1-57b54e8b80a5" (UID: "4e4764c4-743b-4010-a3f1-57b54e8b80a5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:32:50.151731 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.151680    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4764c4-743b-4010-a3f1-57b54e8b80a5-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4e4764c4-743b-4010-a3f1-57b54e8b80a5" (UID: "4e4764c4-743b-4010-a3f1-57b54e8b80a5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:32:50.159941 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.159912    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4764c4-743b-4010-a3f1-57b54e8b80a5-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4e4764c4-743b-4010-a3f1-57b54e8b80a5" (UID: "4e4764c4-743b-4010-a3f1-57b54e8b80a5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:32:50.249435 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.249395    2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e4764c4-743b-4010-a3f1-57b54e8b80a5-ca-trust-extracted\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:32:50.249435 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.249432    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-729ww\" (UniqueName: \"kubernetes.io/projected/4e4764c4-743b-4010-a3f1-57b54e8b80a5-kube-api-access-729ww\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:32:50.249435 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.249443    2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e4764c4-743b-4010-a3f1-57b54e8b80a5-installation-pull-secrets\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:32:50.249653 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.249452    2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e4764c4-743b-4010-a3f1-57b54e8b80a5-registry-tls\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:32:50.249653 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.249460    2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e4764c4-743b-4010-a3f1-57b54e8b80a5-bound-sa-token\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:32:50.249653 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.249470    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e4764c4-743b-4010-a3f1-57b54e8b80a5-trusted-ca\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:32:50.249653 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.249478    2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e4764c4-743b-4010-a3f1-57b54e8b80a5-image-registry-private-configuration\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:32:50.895205 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.895169    2564 generic.go:356] "Generic (PLEG): container finished" podID="4e4764c4-743b-4010-a3f1-57b54e8b80a5" containerID="aa057a48328cd2d650cad06ba333dbd6e170c78bd5fd55abf033aa99db7b20b2" exitCode=0
Jul 17 12:32:50.895677 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.895234    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79777dbb55-7fclv"
Jul 17 12:32:50.895677 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.895250    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79777dbb55-7fclv" event={"ID":"4e4764c4-743b-4010-a3f1-57b54e8b80a5","Type":"ContainerDied","Data":"aa057a48328cd2d650cad06ba333dbd6e170c78bd5fd55abf033aa99db7b20b2"}
Jul 17 12:32:50.895677 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.895302    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79777dbb55-7fclv" event={"ID":"4e4764c4-743b-4010-a3f1-57b54e8b80a5","Type":"ContainerDied","Data":"fcb2d3101ab6c4f398664528ac39599166558ce8709d177c53c4c3237b9e41ec"}
Jul 17 12:32:50.895677 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.895322    2564 scope.go:117] "RemoveContainer" containerID="aa057a48328cd2d650cad06ba333dbd6e170c78bd5fd55abf033aa99db7b20b2"
Jul 17 12:32:50.905236 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.905211    2564 scope.go:117] "RemoveContainer" containerID="ef3c43920f78334dbf5258c279918b7d79863834d9632d51480164bc2c57b924"
Jul 17 12:32:50.914953 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.914935    2564 scope.go:117] "RemoveContainer" containerID="aa057a48328cd2d650cad06ba333dbd6e170c78bd5fd55abf033aa99db7b20b2"
Jul 17 12:32:50.915228 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:32:50.915207    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa057a48328cd2d650cad06ba333dbd6e170c78bd5fd55abf033aa99db7b20b2\": container with ID starting with aa057a48328cd2d650cad06ba333dbd6e170c78bd5fd55abf033aa99db7b20b2 not found: ID does not exist" containerID="aa057a48328cd2d650cad06ba333dbd6e170c78bd5fd55abf033aa99db7b20b2"
Jul 17 12:32:50.915298 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.915242    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa057a48328cd2d650cad06ba333dbd6e170c78bd5fd55abf033aa99db7b20b2"} err="failed to get container status \"aa057a48328cd2d650cad06ba333dbd6e170c78bd5fd55abf033aa99db7b20b2\": rpc error: code = NotFound desc = could not find container \"aa057a48328cd2d650cad06ba333dbd6e170c78bd5fd55abf033aa99db7b20b2\": container with ID starting with aa057a48328cd2d650cad06ba333dbd6e170c78bd5fd55abf033aa99db7b20b2 not found: ID does not exist"
Jul 17 12:32:50.915298 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.915269    2564 scope.go:117] "RemoveContainer" containerID="ef3c43920f78334dbf5258c279918b7d79863834d9632d51480164bc2c57b924"
Jul 17 12:32:50.915565 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:32:50.915543    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef3c43920f78334dbf5258c279918b7d79863834d9632d51480164bc2c57b924\": container with ID starting with ef3c43920f78334dbf5258c279918b7d79863834d9632d51480164bc2c57b924 not found: ID does not exist" containerID="ef3c43920f78334dbf5258c279918b7d79863834d9632d51480164bc2c57b924"
Jul 17 12:32:50.915645 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.915572    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef3c43920f78334dbf5258c279918b7d79863834d9632d51480164bc2c57b924"} err="failed to get container status \"ef3c43920f78334dbf5258c279918b7d79863834d9632d51480164bc2c57b924\": rpc error: code = NotFound desc = could not find container \"ef3c43920f78334dbf5258c279918b7d79863834d9632d51480164bc2c57b924\": container with ID starting with ef3c43920f78334dbf5258c279918b7d79863834d9632d51480164bc2c57b924 not found: ID does not exist"
Jul 17 12:32:50.920264 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.920243    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79777dbb55-7fclv"]
Jul 17 12:32:50.928258 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:50.928237    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-79777dbb55-7fclv"]
Jul 17 12:32:51.016652 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:32:51.016607    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e4764c4-743b-4010-a3f1-57b54e8b80a5" path="/var/lib/kubelet/pods/4e4764c4-743b-4010-a3f1-57b54e8b80a5/volumes"
Jul 17 12:33:08.330049 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:08.330008    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 12:33:08.330618 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:08.330296    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerName="prometheus" containerID="cri-o://d7b6d132a0bb7594e53a09adaeba3b7ebba4cd2f78d383d4cc97c71f899484cf" gracePeriod=600
Jul 17 12:33:08.330618 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:08.330356    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerName="kube-rbac-proxy" containerID="cri-o://b20bd13119c600fa40045b72adde2bdd6ac2c20bc854b571ba8239bb095d7c0b" gracePeriod=600
Jul 17 12:33:08.330618 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:08.330355    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerName="thanos-sidecar" containerID="cri-o://032e09da137bf079916fd1e14e2ecb166d4f9526c6fab1ff1073e346c63f91f6" gracePeriod=600
Jul 17 12:33:08.330618 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:08.330381    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerName="config-reloader" containerID="cri-o://9c3bf91046735e367f7ea162216e9c871f1d00053826ad385e2c1d812a32a8c7" gracePeriod=600
Jul 17 12:33:08.330618 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:08.330407    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerName="kube-rbac-proxy-thanos" containerID="cri-o://6fff370e6c1f236d5f1653db7c34f45da1bdd609b64fa0f3de2b4d8e570f9fcd" gracePeriod=600
Jul 17 12:33:08.330618 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:08.330433    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerName="kube-rbac-proxy-web" containerID="cri-o://1ea58e02b641e26f9f343c7d75eea454bb6a28a3208285fb24b75b5fe3641b7e" gracePeriod=600
Jul 17 12:33:08.972047 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:08.972011    2564 generic.go:356] "Generic (PLEG): container finished" podID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerID="6fff370e6c1f236d5f1653db7c34f45da1bdd609b64fa0f3de2b4d8e570f9fcd" exitCode=0
Jul 17 12:33:08.972047 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:08.972040    2564 generic.go:356] "Generic (PLEG): container finished" podID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerID="b20bd13119c600fa40045b72adde2bdd6ac2c20bc854b571ba8239bb095d7c0b" exitCode=0
Jul 17 12:33:08.972047 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:08.972046    2564 generic.go:356] "Generic (PLEG): container finished" podID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerID="032e09da137bf079916fd1e14e2ecb166d4f9526c6fab1ff1073e346c63f91f6" exitCode=0
Jul 17 12:33:08.972047 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:08.972054    2564 generic.go:356] "Generic (PLEG): container finished" podID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerID="9c3bf91046735e367f7ea162216e9c871f1d00053826ad385e2c1d812a32a8c7" exitCode=0
Jul 17 12:33:08.972310 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:08.972062    2564 generic.go:356] "Generic (PLEG): container finished" podID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerID="d7b6d132a0bb7594e53a09adaeba3b7ebba4cd2f78d383d4cc97c71f899484cf" exitCode=0
Jul 17 12:33:08.972310 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:08.972081    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"174db30d-e76a-49c6-9496-b61e83a4cfae","Type":"ContainerDied","Data":"6fff370e6c1f236d5f1653db7c34f45da1bdd609b64fa0f3de2b4d8e570f9fcd"}
Jul 17 12:33:08.972310 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:08.972152    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"174db30d-e76a-49c6-9496-b61e83a4cfae","Type":"ContainerDied","Data":"b20bd13119c600fa40045b72adde2bdd6ac2c20bc854b571ba8239bb095d7c0b"}
Jul 17 12:33:08.972310 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:08.972178    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"174db30d-e76a-49c6-9496-b61e83a4cfae","Type":"ContainerDied","Data":"032e09da137bf079916fd1e14e2ecb166d4f9526c6fab1ff1073e346c63f91f6"}
Jul 17 12:33:08.972310 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:08.972204    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"174db30d-e76a-49c6-9496-b61e83a4cfae","Type":"ContainerDied","Data":"9c3bf91046735e367f7ea162216e9c871f1d00053826ad385e2c1d812a32a8c7"}
Jul 17 12:33:08.972310 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:08.972228    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"174db30d-e76a-49c6-9496-b61e83a4cfae","Type":"ContainerDied","Data":"d7b6d132a0bb7594e53a09adaeba3b7ebba4cd2f78d383d4cc97c71f899484cf"}
Jul 17 12:33:09.579888 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.579863    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:09.720651 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.720551    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-metrics-client-certs\") pod \"174db30d-e76a-49c6-9496-b61e83a4cfae\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") "
Jul 17 12:33:09.720651 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.720606    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfgpp\" (UniqueName: \"kubernetes.io/projected/174db30d-e76a-49c6-9496-b61e83a4cfae-kube-api-access-qfgpp\") pod \"174db30d-e76a-49c6-9496-b61e83a4cfae\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") "
Jul 17 12:33:09.720651 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.720637    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-configmap-kubelet-serving-ca-bundle\") pod \"174db30d-e76a-49c6-9496-b61e83a4cfae\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") "
Jul 17 12:33:09.720961 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.720660    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"174db30d-e76a-49c6-9496-b61e83a4cfae\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") "
Jul 17 12:33:09.720961 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.720742    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-grpc-tls\") pod \"174db30d-e76a-49c6-9496-b61e83a4cfae\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") "
Jul 17 12:33:09.720961 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.720782    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-prometheus-k8s-rulefiles-1\") pod \"174db30d-e76a-49c6-9496-b61e83a4cfae\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") "
Jul 17 12:33:09.720961 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.720814    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-prometheus-k8s-rulefiles-2\") pod \"174db30d-e76a-49c6-9496-b61e83a4cfae\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") "
Jul 17 12:33:09.720961 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.720840    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-configmap-serving-certs-ca-bundle\") pod \"174db30d-e76a-49c6-9496-b61e83a4cfae\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") "
Jul 17 12:33:09.720961 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.720889    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-prometheus-trusted-ca-bundle\") pod \"174db30d-e76a-49c6-9496-b61e83a4cfae\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") "
Jul 17 12:33:09.720961 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.720922    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-web-config\") pod \"174db30d-e76a-49c6-9496-b61e83a4cfae\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") "
Jul 17 12:33:09.720961 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.720948    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/174db30d-e76a-49c6-9496-b61e83a4cfae-prometheus-k8s-db\") pod \"174db30d-e76a-49c6-9496-b61e83a4cfae\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") "
Jul 17 12:33:09.721343 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.720972    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"174db30d-e76a-49c6-9496-b61e83a4cfae\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") "
Jul 17 12:33:09.721343 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.721011    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-kube-rbac-proxy\") pod \"174db30d-e76a-49c6-9496-b61e83a4cfae\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") "
Jul 17 12:33:09.721343 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.721041    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-configmap-metrics-client-ca\") pod \"174db30d-e76a-49c6-9496-b61e83a4cfae\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") "
Jul 17 12:33:09.721343 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.721072    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-prometheus-k8s-rulefiles-0\") pod \"174db30d-e76a-49c6-9496-b61e83a4cfae\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") "
Jul 17 12:33:09.721343 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.721116    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/174db30d-e76a-49c6-9496-b61e83a4cfae-config-out\") pod \"174db30d-e76a-49c6-9496-b61e83a4cfae\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") "
Jul 17 12:33:09.721343 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.721147    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-prometheus-k8s-tls\") pod \"174db30d-e76a-49c6-9496-b61e83a4cfae\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") "
Jul 17 12:33:09.721343 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.721171    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/174db30d-e76a-49c6-9496-b61e83a4cfae-tls-assets\") pod \"174db30d-e76a-49c6-9496-b61e83a4cfae\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") "
Jul 17 12:33:09.721343 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.721165    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "174db30d-e76a-49c6-9496-b61e83a4cfae" (UID: "174db30d-e76a-49c6-9496-b61e83a4cfae"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:33:09.721343 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.721195    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-config\") pod \"174db30d-e76a-49c6-9496-b61e83a4cfae\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") "
Jul 17 12:33:09.721343 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.721224    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-thanos-prometheus-http-client-file\") pod \"174db30d-e76a-49c6-9496-b61e83a4cfae\" (UID: \"174db30d-e76a-49c6-9496-b61e83a4cfae\") "
Jul 17 12:33:09.721343 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.721273    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "174db30d-e76a-49c6-9496-b61e83a4cfae" (UID: "174db30d-e76a-49c6-9496-b61e83a4cfae"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:33:09.721941 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.721514    2564 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:33:09.721941 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.721537    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:33:09.721941 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.721817    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "174db30d-e76a-49c6-9496-b61e83a4cfae" (UID: "174db30d-e76a-49c6-9496-b61e83a4cfae"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:33:09.725470 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.723819    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "174db30d-e76a-49c6-9496-b61e83a4cfae" (UID: "174db30d-e76a-49c6-9496-b61e83a4cfae"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:33:09.725470 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.724106    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "174db30d-e76a-49c6-9496-b61e83a4cfae" (UID: "174db30d-e76a-49c6-9496-b61e83a4cfae"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:33:09.725470 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.724126    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174db30d-e76a-49c6-9496-b61e83a4cfae-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "174db30d-e76a-49c6-9496-b61e83a4cfae" (UID: "174db30d-e76a-49c6-9496-b61e83a4cfae"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:33:09.725470 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.722524    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "174db30d-e76a-49c6-9496-b61e83a4cfae" (UID: "174db30d-e76a-49c6-9496-b61e83a4cfae"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:33:09.725470 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.724316    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "174db30d-e76a-49c6-9496-b61e83a4cfae" (UID: "174db30d-e76a-49c6-9496-b61e83a4cfae"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:33:09.725470 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.724359    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "174db30d-e76a-49c6-9496-b61e83a4cfae" (UID: "174db30d-e76a-49c6-9496-b61e83a4cfae"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:33:09.725470 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.724386    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "174db30d-e76a-49c6-9496-b61e83a4cfae" (UID: "174db30d-e76a-49c6-9496-b61e83a4cfae"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:33:09.725470 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.724447    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "174db30d-e76a-49c6-9496-b61e83a4cfae" (UID: "174db30d-e76a-49c6-9496-b61e83a4cfae"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:33:09.725470 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.724457    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/174db30d-e76a-49c6-9496-b61e83a4cfae-kube-api-access-qfgpp" (OuterVolumeSpecName: "kube-api-access-qfgpp") pod "174db30d-e76a-49c6-9496-b61e83a4cfae" (UID: "174db30d-e76a-49c6-9496-b61e83a4cfae"). InnerVolumeSpecName "kube-api-access-qfgpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:33:09.727163 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.726404    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "174db30d-e76a-49c6-9496-b61e83a4cfae" (UID: "174db30d-e76a-49c6-9496-b61e83a4cfae"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:33:09.727163 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.726616    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "174db30d-e76a-49c6-9496-b61e83a4cfae" (UID: "174db30d-e76a-49c6-9496-b61e83a4cfae"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:33:09.727163 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.726691    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-config" (OuterVolumeSpecName: "config") pod "174db30d-e76a-49c6-9496-b61e83a4cfae" (UID: "174db30d-e76a-49c6-9496-b61e83a4cfae"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:33:09.727163 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.726769    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "174db30d-e76a-49c6-9496-b61e83a4cfae" (UID: "174db30d-e76a-49c6-9496-b61e83a4cfae"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:33:09.727453 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.727414    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/174db30d-e76a-49c6-9496-b61e83a4cfae-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "174db30d-e76a-49c6-9496-b61e83a4cfae" (UID: "174db30d-e76a-49c6-9496-b61e83a4cfae"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:33:09.727824 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.727785    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "174db30d-e76a-49c6-9496-b61e83a4cfae" (UID: "174db30d-e76a-49c6-9496-b61e83a4cfae"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:33:09.727975 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.727958    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174db30d-e76a-49c6-9496-b61e83a4cfae-config-out" (OuterVolumeSpecName: "config-out") pod "174db30d-e76a-49c6-9496-b61e83a4cfae" (UID: "174db30d-e76a-49c6-9496-b61e83a4cfae"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:33:09.740020 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.739992    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-web-config" (OuterVolumeSpecName: "web-config") pod "174db30d-e76a-49c6-9496-b61e83a4cfae" (UID: "174db30d-e76a-49c6-9496-b61e83a4cfae"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:33:09.822673 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.822624    2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-web-config\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:33:09.822673 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.822669    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/174db30d-e76a-49c6-9496-b61e83a4cfae-prometheus-k8s-db\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:33:09.822673 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.822681    2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:33:09.822906 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.822693    2564 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-kube-rbac-proxy\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:33:09.822906 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.822731    2564 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-configmap-metrics-client-ca\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:33:09.822906 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.822740    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:33:09.822906 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.822749    2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/174db30d-e76a-49c6-9496-b61e83a4cfae-config-out\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:33:09.822906 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.822757    2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:33:09.822906 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.822765    2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/174db30d-e76a-49c6-9496-b61e83a4cfae-tls-assets\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:33:09.822906 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.822774    2564 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-config\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:33:09.822906 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.822782    2564 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:33:09.822906 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.822791    2564 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-metrics-client-certs\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:33:09.822906 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.822799    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfgpp\" (UniqueName: \"kubernetes.io/projected/174db30d-e76a-49c6-9496-b61e83a4cfae-kube-api-access-qfgpp\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:33:09.822906 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.822807    2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:33:09.822906 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.822817    2564 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/174db30d-e76a-49c6-9496-b61e83a4cfae-secret-grpc-tls\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:33:09.822906 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.822825    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:33:09.822906 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.822833    2564 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:33:09.822906 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.822841    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/174db30d-e76a-49c6-9496-b61e83a4cfae-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:33:09.979396 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.979307    2564 generic.go:356] "Generic (PLEG): container finished" podID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerID="1ea58e02b641e26f9f343c7d75eea454bb6a28a3208285fb24b75b5fe3641b7e" exitCode=0
Jul 17 12:33:09.979396 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.979372    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"174db30d-e76a-49c6-9496-b61e83a4cfae","Type":"ContainerDied","Data":"1ea58e02b641e26f9f343c7d75eea454bb6a28a3208285fb24b75b5fe3641b7e"}
Jul 17 12:33:09.979580 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.979400    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:09.979580 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.979476    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"174db30d-e76a-49c6-9496-b61e83a4cfae","Type":"ContainerDied","Data":"9fba51fae4ab29a1eea473ca9543ec1d80dabca4d3e115b328c37417e20c754d"}
Jul 17 12:33:09.979580 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.979523    2564 scope.go:117] "RemoveContainer" containerID="6fff370e6c1f236d5f1653db7c34f45da1bdd609b64fa0f3de2b4d8e570f9fcd"
Jul 17 12:33:09.987302 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.987285    2564 scope.go:117] "RemoveContainer" containerID="b20bd13119c600fa40045b72adde2bdd6ac2c20bc854b571ba8239bb095d7c0b"
Jul 17 12:33:09.993823 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:09.993803    2564 scope.go:117] "RemoveContainer" containerID="1ea58e02b641e26f9f343c7d75eea454bb6a28a3208285fb24b75b5fe3641b7e"
Jul 17 12:33:10.001821 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.001803    2564 scope.go:117] "RemoveContainer" containerID="032e09da137bf079916fd1e14e2ecb166d4f9526c6fab1ff1073e346c63f91f6"
Jul 17 12:33:10.003914 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.003896    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 12:33:10.009076 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.009053    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 12:33:10.009287 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.009270    2564 scope.go:117] "RemoveContainer" containerID="9c3bf91046735e367f7ea162216e9c871f1d00053826ad385e2c1d812a32a8c7"
Jul 17 12:33:10.015868 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.015851    2564 scope.go:117] "RemoveContainer" containerID="d7b6d132a0bb7594e53a09adaeba3b7ebba4cd2f78d383d4cc97c71f899484cf"
Jul 17 12:33:10.023025 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.023007    2564 scope.go:117] "RemoveContainer" containerID="b3e402c723dd83efeeaa197dc0b97f8237d35d6c87a5bd106e0390c11e798f9c"
Jul 17 12:33:10.029815 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.029797    2564 scope.go:117] "RemoveContainer" containerID="6fff370e6c1f236d5f1653db7c34f45da1bdd609b64fa0f3de2b4d8e570f9fcd"
Jul 17 12:33:10.030086 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:33:10.030065    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fff370e6c1f236d5f1653db7c34f45da1bdd609b64fa0f3de2b4d8e570f9fcd\": container with ID starting with 6fff370e6c1f236d5f1653db7c34f45da1bdd609b64fa0f3de2b4d8e570f9fcd not found: ID does not exist" containerID="6fff370e6c1f236d5f1653db7c34f45da1bdd609b64fa0f3de2b4d8e570f9fcd"
Jul 17 12:33:10.030150 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.030095    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fff370e6c1f236d5f1653db7c34f45da1bdd609b64fa0f3de2b4d8e570f9fcd"} err="failed to get container status \"6fff370e6c1f236d5f1653db7c34f45da1bdd609b64fa0f3de2b4d8e570f9fcd\": rpc error: code = NotFound desc = could not find container \"6fff370e6c1f236d5f1653db7c34f45da1bdd609b64fa0f3de2b4d8e570f9fcd\": container with ID starting with 6fff370e6c1f236d5f1653db7c34f45da1bdd609b64fa0f3de2b4d8e570f9fcd not found: ID does not exist"
Jul 17 12:33:10.030150 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.030113    2564 scope.go:117] "RemoveContainer" containerID="b20bd13119c600fa40045b72adde2bdd6ac2c20bc854b571ba8239bb095d7c0b"
Jul 17 12:33:10.030362 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:33:10.030344    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b20bd13119c600fa40045b72adde2bdd6ac2c20bc854b571ba8239bb095d7c0b\": container with ID starting with b20bd13119c600fa40045b72adde2bdd6ac2c20bc854b571ba8239bb095d7c0b not found: ID does not exist" containerID="b20bd13119c600fa40045b72adde2bdd6ac2c20bc854b571ba8239bb095d7c0b"
Jul 17 12:33:10.030406 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.030368    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b20bd13119c600fa40045b72adde2bdd6ac2c20bc854b571ba8239bb095d7c0b"} err="failed to get container status \"b20bd13119c600fa40045b72adde2bdd6ac2c20bc854b571ba8239bb095d7c0b\": rpc error: code = NotFound desc = could not find container \"b20bd13119c600fa40045b72adde2bdd6ac2c20bc854b571ba8239bb095d7c0b\": container with ID starting with b20bd13119c600fa40045b72adde2bdd6ac2c20bc854b571ba8239bb095d7c0b not found: ID does not exist"
Jul 17 12:33:10.030406 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.030384    2564 scope.go:117] "RemoveContainer" containerID="1ea58e02b641e26f9f343c7d75eea454bb6a28a3208285fb24b75b5fe3641b7e"
Jul 17 12:33:10.030621 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:33:10.030603    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ea58e02b641e26f9f343c7d75eea454bb6a28a3208285fb24b75b5fe3641b7e\": container with ID starting with 1ea58e02b641e26f9f343c7d75eea454bb6a28a3208285fb24b75b5fe3641b7e not found: ID does not exist" containerID="1ea58e02b641e26f9f343c7d75eea454bb6a28a3208285fb24b75b5fe3641b7e"
Jul 17 12:33:10.030683 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.030631    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ea58e02b641e26f9f343c7d75eea454bb6a28a3208285fb24b75b5fe3641b7e"} err="failed to get container status \"1ea58e02b641e26f9f343c7d75eea454bb6a28a3208285fb24b75b5fe3641b7e\": rpc error: code = NotFound desc = could not find container \"1ea58e02b641e26f9f343c7d75eea454bb6a28a3208285fb24b75b5fe3641b7e\": container with ID starting with 1ea58e02b641e26f9f343c7d75eea454bb6a28a3208285fb24b75b5fe3641b7e not found: ID does not exist"
Jul 17 12:33:10.030683 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.030654    2564 scope.go:117] "RemoveContainer" containerID="032e09da137bf079916fd1e14e2ecb166d4f9526c6fab1ff1073e346c63f91f6"
Jul 17 12:33:10.030955 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:33:10.030937    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"032e09da137bf079916fd1e14e2ecb166d4f9526c6fab1ff1073e346c63f91f6\": container with ID starting with 032e09da137bf079916fd1e14e2ecb166d4f9526c6fab1ff1073e346c63f91f6 not found: ID does not exist" containerID="032e09da137bf079916fd1e14e2ecb166d4f9526c6fab1ff1073e346c63f91f6"
Jul 17 12:33:10.031026 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.030961    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"032e09da137bf079916fd1e14e2ecb166d4f9526c6fab1ff1073e346c63f91f6"} err="failed to get container status \"032e09da137bf079916fd1e14e2ecb166d4f9526c6fab1ff1073e346c63f91f6\": rpc error: code = NotFound desc = could not find container \"032e09da137bf079916fd1e14e2ecb166d4f9526c6fab1ff1073e346c63f91f6\": container with ID starting with 032e09da137bf079916fd1e14e2ecb166d4f9526c6fab1ff1073e346c63f91f6 not found: ID does not exist"
Jul 17 12:33:10.031026 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.030980    2564 scope.go:117] "RemoveContainer" containerID="9c3bf91046735e367f7ea162216e9c871f1d00053826ad385e2c1d812a32a8c7"
Jul 17 12:33:10.031231 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:33:10.031215    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c3bf91046735e367f7ea162216e9c871f1d00053826ad385e2c1d812a32a8c7\": container with ID starting with 9c3bf91046735e367f7ea162216e9c871f1d00053826ad385e2c1d812a32a8c7 not found: ID does not exist" containerID="9c3bf91046735e367f7ea162216e9c871f1d00053826ad385e2c1d812a32a8c7"
Jul 17 12:33:10.031269 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.031237    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c3bf91046735e367f7ea162216e9c871f1d00053826ad385e2c1d812a32a8c7"} err="failed to get container status \"9c3bf91046735e367f7ea162216e9c871f1d00053826ad385e2c1d812a32a8c7\": rpc error: code = NotFound desc = could not find container \"9c3bf91046735e367f7ea162216e9c871f1d00053826ad385e2c1d812a32a8c7\": container with ID starting with 9c3bf91046735e367f7ea162216e9c871f1d00053826ad385e2c1d812a32a8c7 not found: ID does not exist"
Jul 17 12:33:10.031269 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.031253    2564 scope.go:117] "RemoveContainer" containerID="d7b6d132a0bb7594e53a09adaeba3b7ebba4cd2f78d383d4cc97c71f899484cf"
Jul 17 12:33:10.031476 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:33:10.031460    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7b6d132a0bb7594e53a09adaeba3b7ebba4cd2f78d383d4cc97c71f899484cf\": container with ID starting with d7b6d132a0bb7594e53a09adaeba3b7ebba4cd2f78d383d4cc97c71f899484cf not found: ID does not exist" containerID="d7b6d132a0bb7594e53a09adaeba3b7ebba4cd2f78d383d4cc97c71f899484cf"
Jul 17 12:33:10.031520 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.031481    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7b6d132a0bb7594e53a09adaeba3b7ebba4cd2f78d383d4cc97c71f899484cf"} err="failed to get container status \"d7b6d132a0bb7594e53a09adaeba3b7ebba4cd2f78d383d4cc97c71f899484cf\": rpc error: code = NotFound desc = could not find container \"d7b6d132a0bb7594e53a09adaeba3b7ebba4cd2f78d383d4cc97c71f899484cf\": container with ID starting with d7b6d132a0bb7594e53a09adaeba3b7ebba4cd2f78d383d4cc97c71f899484cf not found: ID does not exist"
Jul 17 12:33:10.031520 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.031495    2564 scope.go:117] "RemoveContainer" containerID="b3e402c723dd83efeeaa197dc0b97f8237d35d6c87a5bd106e0390c11e798f9c"
Jul 17 12:33:10.031773 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:33:10.031719    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3e402c723dd83efeeaa197dc0b97f8237d35d6c87a5bd106e0390c11e798f9c\": container with ID starting with b3e402c723dd83efeeaa197dc0b97f8237d35d6c87a5bd106e0390c11e798f9c not found: ID does not exist" containerID="b3e402c723dd83efeeaa197dc0b97f8237d35d6c87a5bd106e0390c11e798f9c"
Jul 17 12:33:10.031773 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.031738    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3e402c723dd83efeeaa197dc0b97f8237d35d6c87a5bd106e0390c11e798f9c"} err="failed to get container status \"b3e402c723dd83efeeaa197dc0b97f8237d35d6c87a5bd106e0390c11e798f9c\": rpc error: code = NotFound desc = could not find container \"b3e402c723dd83efeeaa197dc0b97f8237d35d6c87a5bd106e0390c11e798f9c\": container with ID starting with b3e402c723dd83efeeaa197dc0b97f8237d35d6c87a5bd106e0390c11e798f9c not found: ID does not exist"
Jul 17 12:33:10.040416 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.040391    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 12:33:10.040661 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.040648    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerName="kube-rbac-proxy"
Jul 17 12:33:10.040729 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.040663    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerName="kube-rbac-proxy"
Jul 17 12:33:10.040729 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.040671    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerName="thanos-sidecar"
Jul 17 12:33:10.040729 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.040676    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerName="thanos-sidecar"
Jul 17 12:33:10.040729 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.040685    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerName="kube-rbac-proxy-web"
Jul 17 12:33:10.040729 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.040692    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerName="kube-rbac-proxy-web"
Jul 17 12:33:10.040729 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.040715    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e4764c4-743b-4010-a3f1-57b54e8b80a5" containerName="registry"
Jul 17 12:33:10.040729 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.040721    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e4764c4-743b-4010-a3f1-57b54e8b80a5" containerName="registry"
Jul 17 12:33:10.040729 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.040727    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerName="prometheus"
Jul 17 12:33:10.040729 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.040732    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerName="prometheus"
Jul 17 12:33:10.041018 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.040739    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerName="kube-rbac-proxy-thanos"
Jul 17 12:33:10.041018 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.040744    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerName="kube-rbac-proxy-thanos"
Jul 17 12:33:10.041018 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.040751    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerName="config-reloader"
Jul 17 12:33:10.041018 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.040755    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerName="config-reloader"
Jul 17 12:33:10.041018 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.040764    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerName="init-config-reloader"
Jul 17 12:33:10.041018 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.040768    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerName="init-config-reloader"
Jul 17 12:33:10.041018 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.040807    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerName="prometheus"
Jul 17 12:33:10.041018 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.040813    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerName="config-reloader"
Jul 17 12:33:10.041018 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.040819    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e4764c4-743b-4010-a3f1-57b54e8b80a5" containerName="registry"
Jul 17 12:33:10.041018 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.040825    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerName="thanos-sidecar"
Jul 17 12:33:10.041018 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.040831    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e4764c4-743b-4010-a3f1-57b54e8b80a5" containerName="registry"
Jul 17 12:33:10.041018 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.040837    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerName="kube-rbac-proxy"
Jul 17 12:33:10.041018 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.040842    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerName="kube-rbac-proxy-thanos"
Jul 17 12:33:10.041018 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.040848    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="174db30d-e76a-49c6-9496-b61e83a4cfae" containerName="kube-rbac-proxy-web"
Jul 17 12:33:10.041018 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.040891    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e4764c4-743b-4010-a3f1-57b54e8b80a5" containerName="registry"
Jul 17 12:33:10.041018 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.040896    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e4764c4-743b-4010-a3f1-57b54e8b80a5" containerName="registry"
Jul 17 12:33:10.045943 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.045923    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.048909 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.048861    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 17 12:33:10.048909 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.048888    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 17 12:33:10.049068 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.048934    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 17 12:33:10.049068 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.048892    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 17 12:33:10.049068 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.049024    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 17 12:33:10.049278 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.049197    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 17 12:33:10.049394 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.049342    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 17 12:33:10.049394 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.049366    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 17 12:33:10.049524 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.049506    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 17 12:33:10.050108 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.050087    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 17 12:33:10.050211 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.050127    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 17 12:33:10.050211 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.050146    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 17 12:33:10.050211 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.050125    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 17 12:33:10.050211 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.050180    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1km0rejukgsc8\""
Jul 17 12:33:10.050393 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.050233    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-g2sbh\""
Jul 17 12:33:10.053361 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.053318    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 17 12:33:10.055171 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.055155    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 17 12:33:10.066393 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.066371    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 12:33:10.227268 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.227231    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a147781b-2b14-4e9c-a69c-46b58bed2ee8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.227268 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.227269    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a147781b-2b14-4e9c-a69c-46b58bed2ee8-web-config\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.227512 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.227289    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a147781b-2b14-4e9c-a69c-46b58bed2ee8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.227512 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.227313    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5v9x\" (UniqueName: \"kubernetes.io/projected/a147781b-2b14-4e9c-a69c-46b58bed2ee8-kube-api-access-p5v9x\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.227512 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.227381    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a147781b-2b14-4e9c-a69c-46b58bed2ee8-config-out\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.227512 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.227402    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a147781b-2b14-4e9c-a69c-46b58bed2ee8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.227512 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.227489    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a147781b-2b14-4e9c-a69c-46b58bed2ee8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.227730 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.227533    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a147781b-2b14-4e9c-a69c-46b58bed2ee8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.227730 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.227590    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a147781b-2b14-4e9c-a69c-46b58bed2ee8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.227730 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.227623    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a147781b-2b14-4e9c-a69c-46b58bed2ee8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.227730 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.227668    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a147781b-2b14-4e9c-a69c-46b58bed2ee8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.227730 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.227718    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a147781b-2b14-4e9c-a69c-46b58bed2ee8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.227886 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.227734    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a147781b-2b14-4e9c-a69c-46b58bed2ee8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.227886 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.227751    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a147781b-2b14-4e9c-a69c-46b58bed2ee8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.227886 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.227780    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a147781b-2b14-4e9c-a69c-46b58bed2ee8-config\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.227886 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.227800    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a147781b-2b14-4e9c-a69c-46b58bed2ee8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.227886 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.227832    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a147781b-2b14-4e9c-a69c-46b58bed2ee8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.227886 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.227866    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a147781b-2b14-4e9c-a69c-46b58bed2ee8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.228057 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.227899    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a147781b-2b14-4e9c-a69c-46b58bed2ee8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.228057 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.227921    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a147781b-2b14-4e9c-a69c-46b58bed2ee8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.328756 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.328683    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a147781b-2b14-4e9c-a69c-46b58bed2ee8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.328908 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.328768    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a147781b-2b14-4e9c-a69c-46b58bed2ee8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.328908 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.328788    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a147781b-2b14-4e9c-a69c-46b58bed2ee8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.328908 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.328810    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a147781b-2b14-4e9c-a69c-46b58bed2ee8-config\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.328908 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.328892    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a147781b-2b14-4e9c-a69c-46b58bed2ee8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.329107 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.328938    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a147781b-2b14-4e9c-a69c-46b58bed2ee8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.329107 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.328962    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a147781b-2b14-4e9c-a69c-46b58bed2ee8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.329107 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.328999    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a147781b-2b14-4e9c-a69c-46b58bed2ee8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.329107 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.329043    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a147781b-2b14-4e9c-a69c-46b58bed2ee8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.329107 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.329078    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a147781b-2b14-4e9c-a69c-46b58bed2ee8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.329107 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.329100    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a147781b-2b14-4e9c-a69c-46b58bed2ee8-web-config\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.329364 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.329130    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a147781b-2b14-4e9c-a69c-46b58bed2ee8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.329364 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.329156    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5v9x\" (UniqueName: \"kubernetes.io/projected/a147781b-2b14-4e9c-a69c-46b58bed2ee8-kube-api-access-p5v9x\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.329364 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.329181    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a147781b-2b14-4e9c-a69c-46b58bed2ee8-config-out\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.329364 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.329204    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a147781b-2b14-4e9c-a69c-46b58bed2ee8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.329364 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.329260    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a147781b-2b14-4e9c-a69c-46b58bed2ee8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.329364 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.329303    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a147781b-2b14-4e9c-a69c-46b58bed2ee8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.329364 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.329334    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a147781b-2b14-4e9c-a69c-46b58bed2ee8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.329753 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.329366    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a147781b-2b14-4e9c-a69c-46b58bed2ee8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.329753 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.329407    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a147781b-2b14-4e9c-a69c-46b58bed2ee8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.329753 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.329671    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a147781b-2b14-4e9c-a69c-46b58bed2ee8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.331394 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.330424    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a147781b-2b14-4e9c-a69c-46b58bed2ee8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.331394 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.330661    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a147781b-2b14-4e9c-a69c-46b58bed2ee8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.331394 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.331019    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a147781b-2b14-4e9c-a69c-46b58bed2ee8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.331394 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.331089    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a147781b-2b14-4e9c-a69c-46b58bed2ee8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.331930 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.331891    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a147781b-2b14-4e9c-a69c-46b58bed2ee8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.332022 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.331972    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a147781b-2b14-4e9c-a69c-46b58bed2ee8-config\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.332334 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.332307    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a147781b-2b14-4e9c-a69c-46b58bed2ee8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.332411 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.332307    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a147781b-2b14-4e9c-a69c-46b58bed2ee8-web-config\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.333192 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.332866    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a147781b-2b14-4e9c-a69c-46b58bed2ee8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.333851 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.333368    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a147781b-2b14-4e9c-a69c-46b58bed2ee8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.333851 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.333811    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a147781b-2b14-4e9c-a69c-46b58bed2ee8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.333851 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.333831    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a147781b-2b14-4e9c-a69c-46b58bed2ee8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.334037 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.333865    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a147781b-2b14-4e9c-a69c-46b58bed2ee8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.334037 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.333949    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a147781b-2b14-4e9c-a69c-46b58bed2ee8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.334172 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.334151    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a147781b-2b14-4e9c-a69c-46b58bed2ee8-config-out\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.334276 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.334258    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a147781b-2b14-4e9c-a69c-46b58bed2ee8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.334367 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.334343    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a147781b-2b14-4e9c-a69c-46b58bed2ee8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.335089 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.335068    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a147781b-2b14-4e9c-a69c-46b58bed2ee8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.340637 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.340621    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5v9x\" (UniqueName: \"kubernetes.io/projected/a147781b-2b14-4e9c-a69c-46b58bed2ee8-kube-api-access-p5v9x\") pod \"prometheus-k8s-0\" (UID: \"a147781b-2b14-4e9c-a69c-46b58bed2ee8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.356627 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.356604    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:10.501528 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.501493    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 12:33:10.505099 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:33:10.505065    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda147781b_2b14_4e9c_a69c_46b58bed2ee8.slice/crio-b183b8c422c296806a2d28c8bcc4433e7dad285556bd63f040789b935354e4dc WatchSource:0}: Error finding container b183b8c422c296806a2d28c8bcc4433e7dad285556bd63f040789b935354e4dc: Status 404 returned error can't find the container with id b183b8c422c296806a2d28c8bcc4433e7dad285556bd63f040789b935354e4dc
Jul 17 12:33:10.989102 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.988993    2564 generic.go:356] "Generic (PLEG): container finished" podID="a147781b-2b14-4e9c-a69c-46b58bed2ee8" containerID="90bba66a1d2b8f00fb1109388b07fe54f228ef3459d342395e2f8dea73dd0145" exitCode=0
Jul 17 12:33:10.989557 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.989147    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a147781b-2b14-4e9c-a69c-46b58bed2ee8","Type":"ContainerDied","Data":"90bba66a1d2b8f00fb1109388b07fe54f228ef3459d342395e2f8dea73dd0145"}
Jul 17 12:33:10.989557 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:10.989323    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a147781b-2b14-4e9c-a69c-46b58bed2ee8","Type":"ContainerStarted","Data":"b183b8c422c296806a2d28c8bcc4433e7dad285556bd63f040789b935354e4dc"}
Jul 17 12:33:11.014561 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:11.014534    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="174db30d-e76a-49c6-9496-b61e83a4cfae" path="/var/lib/kubelet/pods/174db30d-e76a-49c6-9496-b61e83a4cfae/volumes"
Jul 17 12:33:11.996487 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:11.996452    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a147781b-2b14-4e9c-a69c-46b58bed2ee8","Type":"ContainerStarted","Data":"c334d77841d11bbaf17642fe9fe58dd988f5f6416acccb88ad2f6d5a9aa45716"}
Jul 17 12:33:11.996959 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:11.996526    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a147781b-2b14-4e9c-a69c-46b58bed2ee8","Type":"ContainerStarted","Data":"f0850f45b0bf790fdc45ec8a25daeafa6759c66d3147622099c15a38f364d472"}
Jul 17 12:33:11.996959 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:11.996567    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a147781b-2b14-4e9c-a69c-46b58bed2ee8","Type":"ContainerStarted","Data":"0ea135b428463bd5b9154c3aa72a5511a069a755a8c085d24e36bb4f7fe230cc"}
Jul 17 12:33:11.996959 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:11.996606    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a147781b-2b14-4e9c-a69c-46b58bed2ee8","Type":"ContainerStarted","Data":"e4c23804bd8d80b224b291cd9de486ddbac247be4a57fa4e9bf9b1ec5f3c0a02"}
Jul 17 12:33:11.996959 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:11.996662    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a147781b-2b14-4e9c-a69c-46b58bed2ee8","Type":"ContainerStarted","Data":"64e7475ac4ba9cb55b1be057eb5b2937ff4780d5c1c2096399e088a1ba01e91c"}
Jul 17 12:33:11.996959 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:11.996735    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a147781b-2b14-4e9c-a69c-46b58bed2ee8","Type":"ContainerStarted","Data":"7bff47dc2152e3be47b672f87a4e5be414107b6475a8ab89ee82bfcfe4e956f7"}
Jul 17 12:33:12.029237 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:12.029170    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.010377435 podStartE2EDuration="2.029145465s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.76803ms" isStatefulPod=false podCreationTimestamp="2026-07-17 12:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:33:12.027182727 +0000 UTC m=+201.607294767" watchObservedRunningTime="2026-07-17 12:33:12.029145465 +0000 UTC m=+201.609257486"
Jul 17 12:33:15.356899 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:15.356858    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:33:29.557270 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:33:29.557229    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:34:10.357550 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:34:10.357511    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:34:10.373731 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:34:10.373682    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:34:11.231493 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:34:11.231468    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 12:34:26.731994 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:34:26.731913    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:34:50.776873 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:34:50.776846    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f958_7ab622df-86d3-4cc3-aba7-123f4bc0c996/ovn-acl-logging/0.log"
Jul 17 12:34:50.783661 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:34:50.783638    2564 kubelet.go:1648] "Image garbage collection succeeded"
Jul 17 12:34:50.793931 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:34:50.793908    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f958_7ab622df-86d3-4cc3-aba7-123f4bc0c996/ovn-acl-logging/0.log"
Jul 17 12:35:42.553641 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:35:42.553545    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:37:23.611964 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:37:23.611892    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:09.186818 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:38:09.186783    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:15.537320 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:38:15.537284    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:16.395198 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:38:16.395160    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:19.264134 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:38:19.264100    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:19.793579 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:38:19.793545    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:24.636847 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:38:24.636810    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:26.958166 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:38:26.958120    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:28.516661 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:38:28.516623    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:30.225169 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:38:30.225129    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:31.012192 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:38:31.012159    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:31.843129 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:38:31.843083    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:35.068160 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:38:35.068127    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:35.996063 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:38:35.996032    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:46.657691 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:38:46.657609    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:50.800187 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:39:50.800158    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f958_7ab622df-86d3-4cc3-aba7-123f4bc0c996/ovn-acl-logging/0.log"
Jul 17 12:39:50.816609 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:39:50.816583    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f958_7ab622df-86d3-4cc3-aba7-123f4bc0c996/ovn-acl-logging/0.log"
Jul 17 12:40:03.226009 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:40:03.225976    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:42:51.824721 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:42:51.824668    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:42:51.898472 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:42:51.898430    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:09.256755 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:43:09.256693    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:44:50.824297 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:44:50.824225    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f958_7ab622df-86d3-4cc3-aba7-123f4bc0c996/ovn-acl-logging/0.log"
Jul 17 12:44:50.838030 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:44:50.838009    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f958_7ab622df-86d3-4cc3-aba7-123f4bc0c996/ovn-acl-logging/0.log"
Jul 17 12:47:50.562545 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:47:50.562503    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:47:50.627910 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:47:50.627877    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:48:01.347832 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:48:01.347785    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:48:01.470826 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:48:01.470794    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:48:01.801679 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:48:01.801648    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:48:14.349726 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:48:14.349678    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:48:29.376024 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:48:29.375988    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:48:29.466843 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:48:29.466810    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:48:39.533978 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:48:39.533935    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:49:09.632304 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:49:09.632263    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:49:09.733105 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:49:09.733066    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:49:09.959175 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:49:09.959067    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:49:10.058223 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:49:10.058192    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:49:10.167068 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:49:10.167037    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:49:10.264201 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:49:10.264120    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:49:10.408825 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:49:10.408792    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:49:50.847384 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:49:50.847356    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f958_7ab622df-86d3-4cc3-aba7-123f4bc0c996/ovn-acl-logging/0.log"
Jul 17 12:49:50.858964 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:49:50.858941    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f958_7ab622df-86d3-4cc3-aba7-123f4bc0c996/ovn-acl-logging/0.log"
Jul 17 12:50:34.366617 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:50:34.366582    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:50:48.497676 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:50:48.497584    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:50:48.597615 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:50:48.597576    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:50:53.184130 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:50:53.184098    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:50:53.457444 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:50:53.457353    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:50:53.594410 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:50:53.594371    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:50:53.894426 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:50:53.894362    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:50:54.437874 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:50:54.437829    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:50:54.508530 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:50:54.508496    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:50:54.579568 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:50:54.579537    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:50:54.781748 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:50:54.781687    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:50:54.879098 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:50:54.879066    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:50:54.978554 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:50:54.978520    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:50:55.150452 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:50:55.150369    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:50:55.245241 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:50:55.245209    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:52:01.141449 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.140437    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:52:01.206624 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.206590    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6697cdd9c7-mb5lv"]
Jul 17 12:52:01.209861 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.209844    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6697cdd9c7-mb5lv"
Jul 17 12:52:01.213111 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.213089    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 17 12:52:01.214294 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.214270    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-bwzrj\""
Jul 17 12:52:01.214397 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.214313    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 17 12:52:01.214397 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.214277    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 17 12:52:01.221642 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.221619    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6697cdd9c7-mb5lv"]
Jul 17 12:52:01.223454 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.223424    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:52:01.236461 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.236434    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ffcca761-9239-4615-8d5c-93bf1811fe89-cert\") pod \"kserve-controller-manager-6697cdd9c7-mb5lv\" (UID: \"ffcca761-9239-4615-8d5c-93bf1811fe89\") " pod="kserve/kserve-controller-manager-6697cdd9c7-mb5lv"
Jul 17 12:52:01.236598 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.236527    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9z7h\" (UniqueName: \"kubernetes.io/projected/ffcca761-9239-4615-8d5c-93bf1811fe89-kube-api-access-f9z7h\") pod \"kserve-controller-manager-6697cdd9c7-mb5lv\" (UID: \"ffcca761-9239-4615-8d5c-93bf1811fe89\") " pod="kserve/kserve-controller-manager-6697cdd9c7-mb5lv"
Jul 17 12:52:01.246251 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.246224    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-xmj27"]
Jul 17 12:52:01.249414 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.249394    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-xmj27"
Jul 17 12:52:01.252184 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.252167    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 17 12:52:01.252242 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.252207    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-446l4\""
Jul 17 12:52:01.258570 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.258538    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-xmj27"]
Jul 17 12:52:01.298848 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.298812    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:52:01.337030 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.336993    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9z7h\" (UniqueName: \"kubernetes.io/projected/ffcca761-9239-4615-8d5c-93bf1811fe89-kube-api-access-f9z7h\") pod \"kserve-controller-manager-6697cdd9c7-mb5lv\" (UID: \"ffcca761-9239-4615-8d5c-93bf1811fe89\") " pod="kserve/kserve-controller-manager-6697cdd9c7-mb5lv"
Jul 17 12:52:01.337235 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.337065    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mskg\" (UniqueName: \"kubernetes.io/projected/d233aa4a-fed8-47cb-afb6-3513512422ad-kube-api-access-7mskg\") pod \"seaweedfs-64568bcd49-xmj27\" (UID: \"d233aa4a-fed8-47cb-afb6-3513512422ad\") " pod="kserve/seaweedfs-64568bcd49-xmj27"
Jul 17 12:52:01.337235 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.337100    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d233aa4a-fed8-47cb-afb6-3513512422ad-data\") pod \"seaweedfs-64568bcd49-xmj27\" (UID: \"d233aa4a-fed8-47cb-afb6-3513512422ad\") " pod="kserve/seaweedfs-64568bcd49-xmj27"
Jul 17 12:52:01.337235 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.337128    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ffcca761-9239-4615-8d5c-93bf1811fe89-cert\") pod \"kserve-controller-manager-6697cdd9c7-mb5lv\" (UID: \"ffcca761-9239-4615-8d5c-93bf1811fe89\") " pod="kserve/kserve-controller-manager-6697cdd9c7-mb5lv"
Jul 17 12:52:01.337402 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:52:01.337282    2564 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 17 12:52:01.337402 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:52:01.337356    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffcca761-9239-4615-8d5c-93bf1811fe89-cert podName:ffcca761-9239-4615-8d5c-93bf1811fe89 nodeName:}" failed. No retries permitted until 2026-07-17 12:52:01.837333717 +0000 UTC m=+1331.417445729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ffcca761-9239-4615-8d5c-93bf1811fe89-cert") pod "kserve-controller-manager-6697cdd9c7-mb5lv" (UID: "ffcca761-9239-4615-8d5c-93bf1811fe89") : secret "kserve-webhook-server-cert" not found
Jul 17 12:52:01.348327 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.348298    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9z7h\" (UniqueName: \"kubernetes.io/projected/ffcca761-9239-4615-8d5c-93bf1811fe89-kube-api-access-f9z7h\") pod \"kserve-controller-manager-6697cdd9c7-mb5lv\" (UID: \"ffcca761-9239-4615-8d5c-93bf1811fe89\") " pod="kserve/kserve-controller-manager-6697cdd9c7-mb5lv"
Jul 17 12:52:01.432025 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.431935    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:52:01.438132 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.438105    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mskg\" (UniqueName: \"kubernetes.io/projected/d233aa4a-fed8-47cb-afb6-3513512422ad-kube-api-access-7mskg\") pod \"seaweedfs-64568bcd49-xmj27\" (UID: \"d233aa4a-fed8-47cb-afb6-3513512422ad\") " pod="kserve/seaweedfs-64568bcd49-xmj27"
Jul 17 12:52:01.438228 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.438143    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d233aa4a-fed8-47cb-afb6-3513512422ad-data\") pod \"seaweedfs-64568bcd49-xmj27\" (UID: \"d233aa4a-fed8-47cb-afb6-3513512422ad\") " pod="kserve/seaweedfs-64568bcd49-xmj27"
Jul 17 12:52:01.438559 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.438541    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d233aa4a-fed8-47cb-afb6-3513512422ad-data\") pod \"seaweedfs-64568bcd49-xmj27\" (UID: \"d233aa4a-fed8-47cb-afb6-3513512422ad\") " pod="kserve/seaweedfs-64568bcd49-xmj27"
Jul 17 12:52:01.447918 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.447887    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mskg\" (UniqueName: \"kubernetes.io/projected/d233aa4a-fed8-47cb-afb6-3513512422ad-kube-api-access-7mskg\") pod \"seaweedfs-64568bcd49-xmj27\" (UID: \"d233aa4a-fed8-47cb-afb6-3513512422ad\") " pod="kserve/seaweedfs-64568bcd49-xmj27"
Jul 17 12:52:01.535581 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.535548    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:52:01.559342 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.559314    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-xmj27"
Jul 17 12:52:01.688500 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.688412    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-xmj27"]
Jul 17 12:52:01.691114 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:52:01.691086    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd233aa4a_fed8_47cb_afb6_3513512422ad.slice/crio-8c779bdf065ec34885ddd3c10daf12fddb5ecc02650c49456a149ddcdcf2512c WatchSource:0}: Error finding container 8c779bdf065ec34885ddd3c10daf12fddb5ecc02650c49456a149ddcdcf2512c: Status 404 returned error can't find the container with id 8c779bdf065ec34885ddd3c10daf12fddb5ecc02650c49456a149ddcdcf2512c
Jul 17 12:52:01.692412 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.692389    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 12:52:01.842520 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.842477    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ffcca761-9239-4615-8d5c-93bf1811fe89-cert\") pod \"kserve-controller-manager-6697cdd9c7-mb5lv\" (UID: \"ffcca761-9239-4615-8d5c-93bf1811fe89\") " pod="kserve/kserve-controller-manager-6697cdd9c7-mb5lv"
Jul 17 12:52:01.844983 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:01.844963    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ffcca761-9239-4615-8d5c-93bf1811fe89-cert\") pod \"kserve-controller-manager-6697cdd9c7-mb5lv\" (UID: \"ffcca761-9239-4615-8d5c-93bf1811fe89\") " pod="kserve/kserve-controller-manager-6697cdd9c7-mb5lv"
Jul 17 12:52:02.018477 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:02.018393    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-xmj27" event={"ID":"d233aa4a-fed8-47cb-afb6-3513512422ad","Type":"ContainerStarted","Data":"8c779bdf065ec34885ddd3c10daf12fddb5ecc02650c49456a149ddcdcf2512c"}
Jul 17 12:52:02.065422 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:02.065390    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:52:02.121411 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:02.121374    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6697cdd9c7-mb5lv"
Jul 17 12:52:02.162022 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:02.161965    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:52:02.261164 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:02.261133    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:52:02.279305 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:02.279226    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6697cdd9c7-mb5lv"]
Jul 17 12:52:02.375823 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:52:02.375772    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcca761_9239_4615_8d5c_93bf1811fe89.slice/crio-b1ee784b2383ac01ace29bc6a827f7e45baa896140b6d1b8c431007c7f0bd11d WatchSource:0}: Error finding container b1ee784b2383ac01ace29bc6a827f7e45baa896140b6d1b8c431007c7f0bd11d: Status 404 returned error can't find the container with id b1ee784b2383ac01ace29bc6a827f7e45baa896140b6d1b8c431007c7f0bd11d
Jul 17 12:52:03.026169 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:03.026131    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6697cdd9c7-mb5lv" event={"ID":"ffcca761-9239-4615-8d5c-93bf1811fe89","Type":"ContainerStarted","Data":"b1ee784b2383ac01ace29bc6a827f7e45baa896140b6d1b8c431007c7f0bd11d"}
Jul 17 12:52:06.038779 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:06.038739    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-xmj27" event={"ID":"d233aa4a-fed8-47cb-afb6-3513512422ad","Type":"ContainerStarted","Data":"ff132c6432d0aec20cea0241824f14c278cb84f024552e17ad01471429867c07"}
Jul 17 12:52:06.039239 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:06.038817    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-xmj27"
Jul 17 12:52:06.040335 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:06.040304    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6697cdd9c7-mb5lv" event={"ID":"ffcca761-9239-4615-8d5c-93bf1811fe89","Type":"ContainerStarted","Data":"682b051ab7b91fc968d6213d7b3b3fdddda7f7dec024f18d6a8457269f367a52"}
Jul 17 12:52:06.040461 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:06.040444    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6697cdd9c7-mb5lv"
Jul 17 12:52:06.058888 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:06.058824    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-xmj27" podStartSLOduration=0.813727266 podStartE2EDuration="5.058805767s" totalImagesPullingTime="4.245078501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:52:06.058037012 +0000 UTC m=+1335.638149031" watchObservedRunningTime="2026-07-17 12:52:06.058805767 +0000 UTC m=+1335.638917806"
Jul 17 12:52:06.074859 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:06.074802    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6697cdd9c7-mb5lv" podStartSLOduration=1.607334719 podStartE2EDuration="5.074783779s" totalImagesPullingTime="3.46744906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:52:06.074565175 +0000 UTC m=+1335.654677215" watchObservedRunningTime="2026-07-17 12:52:06.074783779 +0000 UTC m=+1335.654895799"
Jul 17 12:52:12.046359 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:12.046327    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-xmj27"
Jul 17 12:52:37.050087 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:37.050051    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6697cdd9c7-mb5lv"
Jul 17 12:52:42.303340 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:42.303307    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6697cdd9c7-mb5lv"]
Jul 17 12:52:42.303779 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:42.303512    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-6697cdd9c7-mb5lv" podUID="ffcca761-9239-4615-8d5c-93bf1811fe89" containerName="manager" containerID="cri-o://682b051ab7b91fc968d6213d7b3b3fdddda7f7dec024f18d6a8457269f367a52" gracePeriod=10
Jul 17 12:52:42.326016 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:42.325989    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6697cdd9c7-mk8zm"]
Jul 17 12:52:42.329097 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:42.329082    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6697cdd9c7-mk8zm"
Jul 17 12:52:42.337227 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:42.337200    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6697cdd9c7-mk8zm"]
Jul 17 12:52:42.393514 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:42.393472    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtdk9\" (UniqueName: \"kubernetes.io/projected/d81d19e0-0bb8-4932-ac57-b8dfb63712df-kube-api-access-gtdk9\") pod \"kserve-controller-manager-6697cdd9c7-mk8zm\" (UID: \"d81d19e0-0bb8-4932-ac57-b8dfb63712df\") " pod="kserve/kserve-controller-manager-6697cdd9c7-mk8zm"
Jul 17 12:52:42.393678 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:42.393528    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d81d19e0-0bb8-4932-ac57-b8dfb63712df-cert\") pod \"kserve-controller-manager-6697cdd9c7-mk8zm\" (UID: \"d81d19e0-0bb8-4932-ac57-b8dfb63712df\") " pod="kserve/kserve-controller-manager-6697cdd9c7-mk8zm"
Jul 17 12:52:42.494433 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:42.494400    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtdk9\" (UniqueName: \"kubernetes.io/projected/d81d19e0-0bb8-4932-ac57-b8dfb63712df-kube-api-access-gtdk9\") pod \"kserve-controller-manager-6697cdd9c7-mk8zm\" (UID: \"d81d19e0-0bb8-4932-ac57-b8dfb63712df\") " pod="kserve/kserve-controller-manager-6697cdd9c7-mk8zm"
Jul 17 12:52:42.494605 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:42.494446    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d81d19e0-0bb8-4932-ac57-b8dfb63712df-cert\") pod \"kserve-controller-manager-6697cdd9c7-mk8zm\" (UID: \"d81d19e0-0bb8-4932-ac57-b8dfb63712df\") " pod="kserve/kserve-controller-manager-6697cdd9c7-mk8zm"
Jul 17 12:52:42.496683 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:42.496660    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d81d19e0-0bb8-4932-ac57-b8dfb63712df-cert\") pod \"kserve-controller-manager-6697cdd9c7-mk8zm\" (UID: \"d81d19e0-0bb8-4932-ac57-b8dfb63712df\") " pod="kserve/kserve-controller-manager-6697cdd9c7-mk8zm"
Jul 17 12:52:42.503500 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:42.503477    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtdk9\" (UniqueName: \"kubernetes.io/projected/d81d19e0-0bb8-4932-ac57-b8dfb63712df-kube-api-access-gtdk9\") pod \"kserve-controller-manager-6697cdd9c7-mk8zm\" (UID: \"d81d19e0-0bb8-4932-ac57-b8dfb63712df\") " pod="kserve/kserve-controller-manager-6697cdd9c7-mk8zm"
Jul 17 12:52:42.536307 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:42.536283    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6697cdd9c7-mb5lv"
Jul 17 12:52:42.595786 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:42.595679    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9z7h\" (UniqueName: \"kubernetes.io/projected/ffcca761-9239-4615-8d5c-93bf1811fe89-kube-api-access-f9z7h\") pod \"ffcca761-9239-4615-8d5c-93bf1811fe89\" (UID: \"ffcca761-9239-4615-8d5c-93bf1811fe89\") "
Jul 17 12:52:42.595786 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:42.595752    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ffcca761-9239-4615-8d5c-93bf1811fe89-cert\") pod \"ffcca761-9239-4615-8d5c-93bf1811fe89\" (UID: \"ffcca761-9239-4615-8d5c-93bf1811fe89\") "
Jul 17 12:52:42.597864 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:42.597840    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffcca761-9239-4615-8d5c-93bf1811fe89-cert" (OuterVolumeSpecName: "cert") pod "ffcca761-9239-4615-8d5c-93bf1811fe89" (UID: "ffcca761-9239-4615-8d5c-93bf1811fe89"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:52:42.597983 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:42.597896    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffcca761-9239-4615-8d5c-93bf1811fe89-kube-api-access-f9z7h" (OuterVolumeSpecName: "kube-api-access-f9z7h") pod "ffcca761-9239-4615-8d5c-93bf1811fe89" (UID: "ffcca761-9239-4615-8d5c-93bf1811fe89"). InnerVolumeSpecName "kube-api-access-f9z7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:52:42.677017 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:42.676979    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6697cdd9c7-mk8zm"
Jul 17 12:52:42.696791 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:42.696766    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9z7h\" (UniqueName: \"kubernetes.io/projected/ffcca761-9239-4615-8d5c-93bf1811fe89-kube-api-access-f9z7h\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:52:42.696791 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:42.696792    2564 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ffcca761-9239-4615-8d5c-93bf1811fe89-cert\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:52:42.803905 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:42.803876    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6697cdd9c7-mk8zm"]
Jul 17 12:52:42.806502 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:52:42.806475    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81d19e0_0bb8_4932_ac57_b8dfb63712df.slice/crio-ae02708add7f81b6298477219a62d3ace705fde830e33f211c3a7674475f1699 WatchSource:0}: Error finding container ae02708add7f81b6298477219a62d3ace705fde830e33f211c3a7674475f1699: Status 404 returned error can't find the container with id ae02708add7f81b6298477219a62d3ace705fde830e33f211c3a7674475f1699
Jul 17 12:52:43.175912 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:43.175876    2564 generic.go:356] "Generic (PLEG): container finished" podID="ffcca761-9239-4615-8d5c-93bf1811fe89" containerID="682b051ab7b91fc968d6213d7b3b3fdddda7f7dec024f18d6a8457269f367a52" exitCode=0
Jul 17 12:52:43.176104 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:43.175934    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6697cdd9c7-mb5lv"
Jul 17 12:52:43.176104 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:43.175927    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6697cdd9c7-mb5lv" event={"ID":"ffcca761-9239-4615-8d5c-93bf1811fe89","Type":"ContainerDied","Data":"682b051ab7b91fc968d6213d7b3b3fdddda7f7dec024f18d6a8457269f367a52"}
Jul 17 12:52:43.176104 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:43.176020    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6697cdd9c7-mb5lv" event={"ID":"ffcca761-9239-4615-8d5c-93bf1811fe89","Type":"ContainerDied","Data":"b1ee784b2383ac01ace29bc6a827f7e45baa896140b6d1b8c431007c7f0bd11d"}
Jul 17 12:52:43.176104 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:43.176050    2564 scope.go:117] "RemoveContainer" containerID="682b051ab7b91fc968d6213d7b3b3fdddda7f7dec024f18d6a8457269f367a52"
Jul 17 12:52:43.177227 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:43.177186    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6697cdd9c7-mk8zm" event={"ID":"d81d19e0-0bb8-4932-ac57-b8dfb63712df","Type":"ContainerStarted","Data":"ae02708add7f81b6298477219a62d3ace705fde830e33f211c3a7674475f1699"}
Jul 17 12:52:43.183906 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:43.183889    2564 scope.go:117] "RemoveContainer" containerID="682b051ab7b91fc968d6213d7b3b3fdddda7f7dec024f18d6a8457269f367a52"
Jul 17 12:52:43.184170 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:52:43.184155    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"682b051ab7b91fc968d6213d7b3b3fdddda7f7dec024f18d6a8457269f367a52\": container with ID starting with 682b051ab7b91fc968d6213d7b3b3fdddda7f7dec024f18d6a8457269f367a52 not found: ID does not exist" containerID="682b051ab7b91fc968d6213d7b3b3fdddda7f7dec024f18d6a8457269f367a52"
Jul 17 12:52:43.184225 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:43.184180    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"682b051ab7b91fc968d6213d7b3b3fdddda7f7dec024f18d6a8457269f367a52"} err="failed to get container status \"682b051ab7b91fc968d6213d7b3b3fdddda7f7dec024f18d6a8457269f367a52\": rpc error: code = NotFound desc = could not find container \"682b051ab7b91fc968d6213d7b3b3fdddda7f7dec024f18d6a8457269f367a52\": container with ID starting with 682b051ab7b91fc968d6213d7b3b3fdddda7f7dec024f18d6a8457269f367a52 not found: ID does not exist"
Jul 17 12:52:43.193445 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:43.193393    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6697cdd9c7-mb5lv"]
Jul 17 12:52:43.194660 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:43.194638    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-6697cdd9c7-mb5lv"]
Jul 17 12:52:44.183617 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:44.183582    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6697cdd9c7-mk8zm" event={"ID":"d81d19e0-0bb8-4932-ac57-b8dfb63712df","Type":"ContainerStarted","Data":"6ac80a9bab99bf0e1b90966d0bf9fea4f6325faeaf941f94f63196219555974b"}
Jul 17 12:52:44.184037 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:44.183747    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6697cdd9c7-mk8zm"
Jul 17 12:52:44.200610 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:44.200557    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6697cdd9c7-mk8zm" podStartSLOduration=1.766151397 podStartE2EDuration="2.200538663s" totalImagesPullingTime="434.387266ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:52:44.199970768 +0000 UTC m=+1373.780082796" watchObservedRunningTime="2026-07-17 12:52:44.200538663 +0000 UTC m=+1373.780650679"
Jul 17 12:52:45.013999 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:45.013959    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffcca761-9239-4615-8d5c-93bf1811fe89" path="/var/lib/kubelet/pods/ffcca761-9239-4615-8d5c-93bf1811fe89/volumes"
Jul 17 12:52:55.963262 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:52:55.963223    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:53:15.192498 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:15.192459    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6697cdd9c7-mk8zm"
Jul 17 12:53:16.135907 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:16.135871    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:53:16.176386 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:16.176325    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-kvtwn"]
Jul 17 12:53:16.177009 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:16.176983    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffcca761-9239-4615-8d5c-93bf1811fe89" containerName="manager"
Jul 17 12:53:16.177009 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:16.177010    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffcca761-9239-4615-8d5c-93bf1811fe89" containerName="manager"
Jul 17 12:53:16.177203 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:16.177146    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffcca761-9239-4615-8d5c-93bf1811fe89" containerName="manager"
Jul 17 12:53:16.180745 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:16.180719    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-kvtwn"
Jul 17 12:53:16.184218 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:16.184196    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 17 12:53:16.184867 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:16.184853    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-c5hhv\""
Jul 17 12:53:16.193002 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:16.192977    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-kvtwn"]
Jul 17 12:53:16.289029 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:16.288988    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/580a2df3-d616-43ab-a3c4-a7ba1da50156-tls-certs\") pod \"model-serving-api-fd65d7d6-kvtwn\" (UID: \"580a2df3-d616-43ab-a3c4-a7ba1da50156\") " pod="kserve/model-serving-api-fd65d7d6-kvtwn"
Jul 17 12:53:16.289197 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:16.289039    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt5zv\" (UniqueName: \"kubernetes.io/projected/580a2df3-d616-43ab-a3c4-a7ba1da50156-kube-api-access-nt5zv\") pod \"model-serving-api-fd65d7d6-kvtwn\" (UID: \"580a2df3-d616-43ab-a3c4-a7ba1da50156\") " pod="kserve/model-serving-api-fd65d7d6-kvtwn"
Jul 17 12:53:16.389776 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:16.389664    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt5zv\" (UniqueName: \"kubernetes.io/projected/580a2df3-d616-43ab-a3c4-a7ba1da50156-kube-api-access-nt5zv\") pod \"model-serving-api-fd65d7d6-kvtwn\" (UID: \"580a2df3-d616-43ab-a3c4-a7ba1da50156\") " pod="kserve/model-serving-api-fd65d7d6-kvtwn"
Jul 17 12:53:16.389928 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:16.389781    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/580a2df3-d616-43ab-a3c4-a7ba1da50156-tls-certs\") pod \"model-serving-api-fd65d7d6-kvtwn\" (UID: \"580a2df3-d616-43ab-a3c4-a7ba1da50156\") " pod="kserve/model-serving-api-fd65d7d6-kvtwn"
Jul 17 12:53:16.392147 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:16.392117    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/580a2df3-d616-43ab-a3c4-a7ba1da50156-tls-certs\") pod \"model-serving-api-fd65d7d6-kvtwn\" (UID: \"580a2df3-d616-43ab-a3c4-a7ba1da50156\") " pod="kserve/model-serving-api-fd65d7d6-kvtwn"
Jul 17 12:53:16.401977 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:16.401948    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt5zv\" (UniqueName: \"kubernetes.io/projected/580a2df3-d616-43ab-a3c4-a7ba1da50156-kube-api-access-nt5zv\") pod \"model-serving-api-fd65d7d6-kvtwn\" (UID: \"580a2df3-d616-43ab-a3c4-a7ba1da50156\") " pod="kserve/model-serving-api-fd65d7d6-kvtwn"
Jul 17 12:53:16.490917 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:16.490874    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-kvtwn"
Jul 17 12:53:16.621130 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:16.621089    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-kvtwn"]
Jul 17 12:53:16.625566 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:53:16.625525    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod580a2df3_d616_43ab_a3c4_a7ba1da50156.slice/crio-7161908c426967e819be396279d226ec5130cbef3101ce63de3e8f4730aad4b7 WatchSource:0}: Error finding container 7161908c426967e819be396279d226ec5130cbef3101ce63de3e8f4730aad4b7: Status 404 returned error can't find the container with id 7161908c426967e819be396279d226ec5130cbef3101ce63de3e8f4730aad4b7
Jul 17 12:53:17.290065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:17.290029    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:53:17.307214 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:17.307178    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-kvtwn" event={"ID":"580a2df3-d616-43ab-a3c4-a7ba1da50156","Type":"ContainerStarted","Data":"7161908c426967e819be396279d226ec5130cbef3101ce63de3e8f4730aad4b7"}
Jul 17 12:53:18.312613 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:18.312580    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-kvtwn" event={"ID":"580a2df3-d616-43ab-a3c4-a7ba1da50156","Type":"ContainerStarted","Data":"6ebe3bee79fdd4437cde3223b113e6b43d3442f63a40c66d3ddaa93b9fda5985"}
Jul 17 12:53:18.313000 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:18.312737    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-kvtwn"
Jul 17 12:53:18.333612 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:18.333551    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-kvtwn" podStartSLOduration=0.728967427 podStartE2EDuration="2.33353136s" totalImagesPullingTime="1.604563933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:53:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:53:18.332406099 +0000 UTC m=+1407.912518139" watchObservedRunningTime="2026-07-17 12:53:18.33353136 +0000 UTC m=+1407.913643380"
Jul 17 12:53:29.319988 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:29.319956    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-kvtwn"
Jul 17 12:53:31.229355 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:31.229315    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-hwtrx"]
Jul 17 12:53:31.232630 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:31.232610    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-hwtrx"
Jul 17 12:53:31.240612 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:31.240587    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-hwtrx"]
Jul 17 12:53:31.318515 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:31.318477    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb94f\" (UniqueName: \"kubernetes.io/projected/5552b3ce-f442-4af3-a184-1b0d8e7466d4-kube-api-access-pb94f\") pod \"s3-init-hwtrx\" (UID: \"5552b3ce-f442-4af3-a184-1b0d8e7466d4\") " pod="kserve/s3-init-hwtrx"
Jul 17 12:53:31.318691 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:31.318537    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5552b3ce-f442-4af3-a184-1b0d8e7466d4-model-cache\") pod \"s3-init-hwtrx\" (UID: \"5552b3ce-f442-4af3-a184-1b0d8e7466d4\") " pod="kserve/s3-init-hwtrx"
Jul 17 12:53:31.419384 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:31.419349    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb94f\" (UniqueName: \"kubernetes.io/projected/5552b3ce-f442-4af3-a184-1b0d8e7466d4-kube-api-access-pb94f\") pod \"s3-init-hwtrx\" (UID: \"5552b3ce-f442-4af3-a184-1b0d8e7466d4\") " pod="kserve/s3-init-hwtrx"
Jul 17 12:53:31.419569 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:31.419404    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5552b3ce-f442-4af3-a184-1b0d8e7466d4-model-cache\") pod \"s3-init-hwtrx\" (UID: \"5552b3ce-f442-4af3-a184-1b0d8e7466d4\") " pod="kserve/s3-init-hwtrx"
Jul 17 12:53:31.419800 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:31.419781    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5552b3ce-f442-4af3-a184-1b0d8e7466d4-model-cache\") pod \"s3-init-hwtrx\" (UID: \"5552b3ce-f442-4af3-a184-1b0d8e7466d4\") " pod="kserve/s3-init-hwtrx"
Jul 17 12:53:31.428788 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:31.428765    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb94f\" (UniqueName: \"kubernetes.io/projected/5552b3ce-f442-4af3-a184-1b0d8e7466d4-kube-api-access-pb94f\") pod \"s3-init-hwtrx\" (UID: \"5552b3ce-f442-4af3-a184-1b0d8e7466d4\") " pod="kserve/s3-init-hwtrx"
Jul 17 12:53:31.548989 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:31.548961    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-hwtrx"
Jul 17 12:53:31.680333 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:31.680306    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-hwtrx"]
Jul 17 12:53:31.683054 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:53:31.683004    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5552b3ce_f442_4af3_a184_1b0d8e7466d4.slice/crio-6138ecf8c4b1c73a14ef156bfd572f357bed2524dd7c79b847a7a68c191e10b9 WatchSource:0}: Error finding container 6138ecf8c4b1c73a14ef156bfd572f357bed2524dd7c79b847a7a68c191e10b9: Status 404 returned error can't find the container with id 6138ecf8c4b1c73a14ef156bfd572f357bed2524dd7c79b847a7a68c191e10b9
Jul 17 12:53:32.362243 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:32.362207    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hwtrx" event={"ID":"5552b3ce-f442-4af3-a184-1b0d8e7466d4","Type":"ContainerStarted","Data":"6138ecf8c4b1c73a14ef156bfd572f357bed2524dd7c79b847a7a68c191e10b9"}
Jul 17 12:53:35.374767 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:35.374676    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hwtrx" event={"ID":"5552b3ce-f442-4af3-a184-1b0d8e7466d4","Type":"ContainerStarted","Data":"d4e3b74ee396b41952fc5dd8efe690dccda66d5793bac02fe98e8a25a802bfd5"}
Jul 17 12:53:36.380093 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:36.380060    2564 generic.go:356] "Generic (PLEG): container finished" podID="5552b3ce-f442-4af3-a184-1b0d8e7466d4" containerID="d4e3b74ee396b41952fc5dd8efe690dccda66d5793bac02fe98e8a25a802bfd5" exitCode=0
Jul 17 12:53:36.380093 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:36.380097    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hwtrx" event={"ID":"5552b3ce-f442-4af3-a184-1b0d8e7466d4","Type":"ContainerDied","Data":"d4e3b74ee396b41952fc5dd8efe690dccda66d5793bac02fe98e8a25a802bfd5"}
Jul 17 12:53:41.404784 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:41.404753    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hwtrx" event={"ID":"5552b3ce-f442-4af3-a184-1b0d8e7466d4","Type":"ContainerStarted","Data":"4871d02d6a7a25eaabad1793e26c8f561cfa5fffc0bb825145eb01d543c30192"}
Jul 17 12:53:41.422129 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:53:41.422074    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-hwtrx" podStartSLOduration=1.3385764359999999 podStartE2EDuration="10.422057727s" totalImagesPullingTime="7.933689441s" totalInitContainerRuntime="1.14979185s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:53:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:53:41.420522539 +0000 UTC m=+1431.000634556" watchObservedRunningTime="2026-07-17 12:53:41.422057727 +0000 UTC m=+1431.002169778"
Jul 17 12:54:46.651152 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:46.651115    2564 generic.go:356] "Generic (PLEG): container finished" podID="5552b3ce-f442-4af3-a184-1b0d8e7466d4" containerID="4871d02d6a7a25eaabad1793e26c8f561cfa5fffc0bb825145eb01d543c30192" exitCode=0
Jul 17 12:54:46.651565 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:46.651188    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hwtrx" event={"ID":"5552b3ce-f442-4af3-a184-1b0d8e7466d4","Type":"ContainerDied","Data":"4871d02d6a7a25eaabad1793e26c8f561cfa5fffc0bb825145eb01d543c30192"}
Jul 17 12:54:47.773187 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:47.773157    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-hwtrx"
Jul 17 12:54:47.858799 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:47.858764    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5552b3ce-f442-4af3-a184-1b0d8e7466d4-model-cache\") pod \"5552b3ce-f442-4af3-a184-1b0d8e7466d4\" (UID: \"5552b3ce-f442-4af3-a184-1b0d8e7466d4\") "
Jul 17 12:54:47.859001 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:47.858904    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pb94f\" (UniqueName: \"kubernetes.io/projected/5552b3ce-f442-4af3-a184-1b0d8e7466d4-kube-api-access-pb94f\") pod \"5552b3ce-f442-4af3-a184-1b0d8e7466d4\" (UID: \"5552b3ce-f442-4af3-a184-1b0d8e7466d4\") "
Jul 17 12:54:47.860964 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:47.860936    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5552b3ce-f442-4af3-a184-1b0d8e7466d4-kube-api-access-pb94f" (OuterVolumeSpecName: "kube-api-access-pb94f") pod "5552b3ce-f442-4af3-a184-1b0d8e7466d4" (UID: "5552b3ce-f442-4af3-a184-1b0d8e7466d4"). InnerVolumeSpecName "kube-api-access-pb94f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:54:47.877925 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:47.877887    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5552b3ce-f442-4af3-a184-1b0d8e7466d4-model-cache" (OuterVolumeSpecName: "model-cache") pod "5552b3ce-f442-4af3-a184-1b0d8e7466d4" (UID: "5552b3ce-f442-4af3-a184-1b0d8e7466d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:54:47.959509 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:47.959424    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb94f\" (UniqueName: \"kubernetes.io/projected/5552b3ce-f442-4af3-a184-1b0d8e7466d4-kube-api-access-pb94f\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:54:47.959509 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:47.959458    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5552b3ce-f442-4af3-a184-1b0d8e7466d4-model-cache\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:54:48.659961 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:48.659924    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hwtrx" event={"ID":"5552b3ce-f442-4af3-a184-1b0d8e7466d4","Type":"ContainerDied","Data":"6138ecf8c4b1c73a14ef156bfd572f357bed2524dd7c79b847a7a68c191e10b9"}
Jul 17 12:54:48.659961 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:48.659950    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-hwtrx"
Jul 17 12:54:48.659961 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:48.659967    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6138ecf8c4b1c73a14ef156bfd572f357bed2524dd7c79b847a7a68c191e10b9"
Jul 17 12:54:49.402319 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:49.402281    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-hr2z7"]
Jul 17 12:54:49.402691 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:49.402585    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5552b3ce-f442-4af3-a184-1b0d8e7466d4" containerName="s3-init"
Jul 17 12:54:49.402691 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:49.402598    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5552b3ce-f442-4af3-a184-1b0d8e7466d4" containerName="s3-init"
Jul 17 12:54:49.402691 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:49.402622    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5552b3ce-f442-4af3-a184-1b0d8e7466d4" containerName="download-hf-model"
Jul 17 12:54:49.402691 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:49.402629    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5552b3ce-f442-4af3-a184-1b0d8e7466d4" containerName="download-hf-model"
Jul 17 12:54:49.402842 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:49.402675    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5552b3ce-f442-4af3-a184-1b0d8e7466d4" containerName="s3-init"
Jul 17 12:54:49.405839 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:49.405819    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hr2z7"
Jul 17 12:54:49.408506 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:49.408488    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 17 12:54:49.415014 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:49.414993    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-hr2z7"]
Jul 17 12:54:49.432886 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:49.432854    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:54:49.472421 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:49.472383    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8589812b-8211-4676-8d32-28958d8900ec-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-hr2z7\" (UID: \"8589812b-8211-4676-8d32-28958d8900ec\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hr2z7"
Jul 17 12:54:49.472584 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:49.472433    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cn2b\" (UniqueName: \"kubernetes.io/projected/8589812b-8211-4676-8d32-28958d8900ec-kube-api-access-9cn2b\") pod \"seaweedfs-tls-custom-6644bf5d6c-hr2z7\" (UID: \"8589812b-8211-4676-8d32-28958d8900ec\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hr2z7"
Jul 17 12:54:49.573295 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:49.573257    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cn2b\" (UniqueName: \"kubernetes.io/projected/8589812b-8211-4676-8d32-28958d8900ec-kube-api-access-9cn2b\") pod \"seaweedfs-tls-custom-6644bf5d6c-hr2z7\" (UID: \"8589812b-8211-4676-8d32-28958d8900ec\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hr2z7"
Jul 17 12:54:49.573440 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:49.573353    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8589812b-8211-4676-8d32-28958d8900ec-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-hr2z7\" (UID: \"8589812b-8211-4676-8d32-28958d8900ec\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hr2z7"
Jul 17 12:54:49.573772 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:49.573755    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8589812b-8211-4676-8d32-28958d8900ec-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-hr2z7\" (UID: \"8589812b-8211-4676-8d32-28958d8900ec\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hr2z7"
Jul 17 12:54:49.583243 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:49.583209    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cn2b\" (UniqueName: \"kubernetes.io/projected/8589812b-8211-4676-8d32-28958d8900ec-kube-api-access-9cn2b\") pod \"seaweedfs-tls-custom-6644bf5d6c-hr2z7\" (UID: \"8589812b-8211-4676-8d32-28958d8900ec\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hr2z7"
Jul 17 12:54:49.715420 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:49.715329    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hr2z7"
Jul 17 12:54:49.843717 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:49.843679    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-hr2z7"]
Jul 17 12:54:49.845819 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:54:49.845787    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8589812b_8211_4676_8d32_28958d8900ec.slice/crio-4a78f26fdc7128688ef1a52da1f48f82b224def228e80a28b646bd03b2d5a73a WatchSource:0}: Error finding container 4a78f26fdc7128688ef1a52da1f48f82b224def228e80a28b646bd03b2d5a73a: Status 404 returned error can't find the container with id 4a78f26fdc7128688ef1a52da1f48f82b224def228e80a28b646bd03b2d5a73a
Jul 17 12:54:50.669366 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:50.669335    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hr2z7" event={"ID":"8589812b-8211-4676-8d32-28958d8900ec","Type":"ContainerStarted","Data":"65a7f407dda1677eda29094f48fc17b5c530e5d90b41d7b3319fa63c680a4455"}
Jul 17 12:54:50.669822 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:50.669384    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hr2z7" event={"ID":"8589812b-8211-4676-8d32-28958d8900ec","Type":"ContainerStarted","Data":"4a78f26fdc7128688ef1a52da1f48f82b224def228e80a28b646bd03b2d5a73a"}
Jul 17 12:54:50.687398 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:50.687326    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hr2z7" podStartSLOduration=1.393461291 podStartE2EDuration="1.687306288s" totalImagesPullingTime="293.844997ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:54:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:54:50.685886678 +0000 UTC m=+1500.265998718" watchObservedRunningTime="2026-07-17 12:54:50.687306288 +0000 UTC m=+1500.267418306"
Jul 17 12:54:50.876326 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:50.876300    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f958_7ab622df-86d3-4cc3-aba7-123f4bc0c996/ovn-acl-logging/0.log"
Jul 17 12:54:50.884354 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:50.884329    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f958_7ab622df-86d3-4cc3-aba7-123f4bc0c996/ovn-acl-logging/0.log"
Jul 17 12:54:52.091408 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:52.091365    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-hr2z7"]
Jul 17 12:54:52.676899 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:52.676864    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hr2z7" podUID="8589812b-8211-4676-8d32-28958d8900ec" containerName="seaweedfs-tls-custom" containerID="cri-o://65a7f407dda1677eda29094f48fc17b5c530e5d90b41d7b3319fa63c680a4455" gracePeriod=30
Jul 17 12:54:53.912459 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:53.912435    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hr2z7"
Jul 17 12:54:54.010144 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:54.010063    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8589812b-8211-4676-8d32-28958d8900ec-data\") pod \"8589812b-8211-4676-8d32-28958d8900ec\" (UID: \"8589812b-8211-4676-8d32-28958d8900ec\") "
Jul 17 12:54:54.010144 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:54.010125    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cn2b\" (UniqueName: \"kubernetes.io/projected/8589812b-8211-4676-8d32-28958d8900ec-kube-api-access-9cn2b\") pod \"8589812b-8211-4676-8d32-28958d8900ec\" (UID: \"8589812b-8211-4676-8d32-28958d8900ec\") "
Jul 17 12:54:54.011305 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:54.011274    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8589812b-8211-4676-8d32-28958d8900ec-data" (OuterVolumeSpecName: "data") pod "8589812b-8211-4676-8d32-28958d8900ec" (UID: "8589812b-8211-4676-8d32-28958d8900ec"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:54:54.012150 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:54.012128    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8589812b-8211-4676-8d32-28958d8900ec-kube-api-access-9cn2b" (OuterVolumeSpecName: "kube-api-access-9cn2b") pod "8589812b-8211-4676-8d32-28958d8900ec" (UID: "8589812b-8211-4676-8d32-28958d8900ec"). InnerVolumeSpecName "kube-api-access-9cn2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:54:54.110916 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:54.110875    2564 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8589812b-8211-4676-8d32-28958d8900ec-data\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:54:54.110916 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:54.110907    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cn2b\" (UniqueName: \"kubernetes.io/projected/8589812b-8211-4676-8d32-28958d8900ec-kube-api-access-9cn2b\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:54:54.687291 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:54.687257    2564 generic.go:356] "Generic (PLEG): container finished" podID="8589812b-8211-4676-8d32-28958d8900ec" containerID="65a7f407dda1677eda29094f48fc17b5c530e5d90b41d7b3319fa63c680a4455" exitCode=0
Jul 17 12:54:54.687291 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:54.687294    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hr2z7" event={"ID":"8589812b-8211-4676-8d32-28958d8900ec","Type":"ContainerDied","Data":"65a7f407dda1677eda29094f48fc17b5c530e5d90b41d7b3319fa63c680a4455"}
Jul 17 12:54:54.687571 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:54.687298    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hr2z7"
Jul 17 12:54:54.687571 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:54.687336    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hr2z7" event={"ID":"8589812b-8211-4676-8d32-28958d8900ec","Type":"ContainerDied","Data":"4a78f26fdc7128688ef1a52da1f48f82b224def228e80a28b646bd03b2d5a73a"}
Jul 17 12:54:54.687571 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:54.687356    2564 scope.go:117] "RemoveContainer" containerID="65a7f407dda1677eda29094f48fc17b5c530e5d90b41d7b3319fa63c680a4455"
Jul 17 12:54:54.696542 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:54.696516    2564 scope.go:117] "RemoveContainer" containerID="65a7f407dda1677eda29094f48fc17b5c530e5d90b41d7b3319fa63c680a4455"
Jul 17 12:54:54.696851 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:54:54.696832    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65a7f407dda1677eda29094f48fc17b5c530e5d90b41d7b3319fa63c680a4455\": container with ID starting with 65a7f407dda1677eda29094f48fc17b5c530e5d90b41d7b3319fa63c680a4455 not found: ID does not exist" containerID="65a7f407dda1677eda29094f48fc17b5c530e5d90b41d7b3319fa63c680a4455"
Jul 17 12:54:54.696918 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:54.696862    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65a7f407dda1677eda29094f48fc17b5c530e5d90b41d7b3319fa63c680a4455"} err="failed to get container status \"65a7f407dda1677eda29094f48fc17b5c530e5d90b41d7b3319fa63c680a4455\": rpc error: code = NotFound desc = could not find container \"65a7f407dda1677eda29094f48fc17b5c530e5d90b41d7b3319fa63c680a4455\": container with ID starting with 65a7f407dda1677eda29094f48fc17b5c530e5d90b41d7b3319fa63c680a4455 not found: ID does not exist"
Jul 17 12:54:54.716373 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:54.716344    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-hr2z7"]
Jul 17 12:54:54.720039 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:54.720014    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-hr2z7"]
Jul 17 12:54:54.754992 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:54.754957    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-htg94"]
Jul 17 12:54:54.755248 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:54.755236    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8589812b-8211-4676-8d32-28958d8900ec" containerName="seaweedfs-tls-custom"
Jul 17 12:54:54.755289 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:54.755249    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8589812b-8211-4676-8d32-28958d8900ec" containerName="seaweedfs-tls-custom"
Jul 17 12:54:54.755326 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:54.755303    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8589812b-8211-4676-8d32-28958d8900ec" containerName="seaweedfs-tls-custom"
Jul 17 12:54:54.759627 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:54.759609    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-htg94"
Jul 17 12:54:54.762364 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:54.762339    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 17 12:54:54.762497 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:54.762369    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Jul 17 12:54:54.766278 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:54.766254    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-htg94"]
Jul 17 12:54:54.917839 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:54.917800    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/9cb42b44-51b0-4a13-8b7d-45069950b35f-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-htg94\" (UID: \"9cb42b44-51b0-4a13-8b7d-45069950b35f\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-htg94"
Jul 17 12:54:54.918234 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:54.917860    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5qsr\" (UniqueName: \"kubernetes.io/projected/9cb42b44-51b0-4a13-8b7d-45069950b35f-kube-api-access-z5qsr\") pod \"seaweedfs-tls-custom-8b44d767d-htg94\" (UID: \"9cb42b44-51b0-4a13-8b7d-45069950b35f\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-htg94"
Jul 17 12:54:54.918234 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:54.917959    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9cb42b44-51b0-4a13-8b7d-45069950b35f-data\") pod \"seaweedfs-tls-custom-8b44d767d-htg94\" (UID: \"9cb42b44-51b0-4a13-8b7d-45069950b35f\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-htg94"
Jul 17 12:54:55.013621 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:55.013544    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8589812b-8211-4676-8d32-28958d8900ec" path="/var/lib/kubelet/pods/8589812b-8211-4676-8d32-28958d8900ec/volumes"
Jul 17 12:54:55.018976 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:55.018955    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/9cb42b44-51b0-4a13-8b7d-45069950b35f-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-htg94\" (UID: \"9cb42b44-51b0-4a13-8b7d-45069950b35f\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-htg94"
Jul 17 12:54:55.019049 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:55.019003    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5qsr\" (UniqueName: \"kubernetes.io/projected/9cb42b44-51b0-4a13-8b7d-45069950b35f-kube-api-access-z5qsr\") pod \"seaweedfs-tls-custom-8b44d767d-htg94\" (UID: \"9cb42b44-51b0-4a13-8b7d-45069950b35f\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-htg94"
Jul 17 12:54:55.019049 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:55.019036    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9cb42b44-51b0-4a13-8b7d-45069950b35f-data\") pod \"seaweedfs-tls-custom-8b44d767d-htg94\" (UID: \"9cb42b44-51b0-4a13-8b7d-45069950b35f\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-htg94"
Jul 17 12:54:55.019348 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:55.019331    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9cb42b44-51b0-4a13-8b7d-45069950b35f-data\") pod \"seaweedfs-tls-custom-8b44d767d-htg94\" (UID: \"9cb42b44-51b0-4a13-8b7d-45069950b35f\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-htg94"
Jul 17 12:54:55.021517 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:55.021495    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/9cb42b44-51b0-4a13-8b7d-45069950b35f-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-htg94\" (UID: \"9cb42b44-51b0-4a13-8b7d-45069950b35f\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-htg94"
Jul 17 12:54:55.028550 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:55.028523    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5qsr\" (UniqueName: \"kubernetes.io/projected/9cb42b44-51b0-4a13-8b7d-45069950b35f-kube-api-access-z5qsr\") pod \"seaweedfs-tls-custom-8b44d767d-htg94\" (UID: \"9cb42b44-51b0-4a13-8b7d-45069950b35f\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-htg94"
Jul 17 12:54:55.069375 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:55.069324    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-htg94"
Jul 17 12:54:55.198988 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:55.198968    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-htg94"]
Jul 17 12:54:55.200609 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:54:55.200583    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb42b44_51b0_4a13_8b7d_45069950b35f.slice/crio-f584da9a96685fa110fb553f74d32aa65798355b02f51c006a13a6e23c3c3d75 WatchSource:0}: Error finding container f584da9a96685fa110fb553f74d32aa65798355b02f51c006a13a6e23c3c3d75: Status 404 returned error can't find the container with id f584da9a96685fa110fb553f74d32aa65798355b02f51c006a13a6e23c3c3d75
Jul 17 12:54:55.693116 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:55.693077    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-htg94" event={"ID":"9cb42b44-51b0-4a13-8b7d-45069950b35f","Type":"ContainerStarted","Data":"99d2268a192b41170e592345ae33e1af6114245efa2a7d560e29e1ebc4b9241d"}
Jul 17 12:54:55.693318 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:55.693148    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-htg94" event={"ID":"9cb42b44-51b0-4a13-8b7d-45069950b35f","Type":"ContainerStarted","Data":"f584da9a96685fa110fb553f74d32aa65798355b02f51c006a13a6e23c3c3d75"}
Jul 17 12:54:55.713178 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:55.713126    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-htg94" podStartSLOduration=1.431164428 podStartE2EDuration="1.713112476s" totalImagesPullingTime="281.948048ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:54:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:54:55.711842561 +0000 UTC m=+1505.291954578" watchObservedRunningTime="2026-07-17 12:54:55.713112476 +0000 UTC m=+1505.293224495"
Jul 17 12:54:56.057337 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:56.057304    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-6br26"]
Jul 17 12:54:56.060615 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:56.060594    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6br26"
Jul 17 12:54:56.078010 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:56.077976    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-6br26"]
Jul 17 12:54:56.127899 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:56.127853    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gcgg\" (UniqueName: \"kubernetes.io/projected/c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae-kube-api-access-6gcgg\") pod \"s3-tls-init-custom-6br26\" (UID: \"c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae\") " pod="kserve/s3-tls-init-custom-6br26"
Jul 17 12:54:56.128078 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:56.127922    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae-model-cache\") pod \"s3-tls-init-custom-6br26\" (UID: \"c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae\") " pod="kserve/s3-tls-init-custom-6br26"
Jul 17 12:54:56.229029 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:56.228992    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gcgg\" (UniqueName: \"kubernetes.io/projected/c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae-kube-api-access-6gcgg\") pod \"s3-tls-init-custom-6br26\" (UID: \"c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae\") " pod="kserve/s3-tls-init-custom-6br26"
Jul 17 12:54:56.229203 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:56.229049    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae-model-cache\") pod \"s3-tls-init-custom-6br26\" (UID: \"c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae\") " pod="kserve/s3-tls-init-custom-6br26"
Jul 17 12:54:56.229438 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:56.229418    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae-model-cache\") pod \"s3-tls-init-custom-6br26\" (UID: \"c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae\") " pod="kserve/s3-tls-init-custom-6br26"
Jul 17 12:54:56.239754 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:56.239695    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gcgg\" (UniqueName: \"kubernetes.io/projected/c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae-kube-api-access-6gcgg\") pod \"s3-tls-init-custom-6br26\" (UID: \"c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae\") " pod="kserve/s3-tls-init-custom-6br26"
Jul 17 12:54:56.388240 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:56.388168    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6br26"
Jul 17 12:54:56.524454 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:56.524425    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-6br26"]
Jul 17 12:54:56.525670 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:54:56.525642    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5c37dbb_0cf4_4ceb_87b8_a331a4a7eaae.slice/crio-b82dfa6b88731e39d5e4538071f0082b92768a854e6d43e47a708c932b6769e9 WatchSource:0}: Error finding container b82dfa6b88731e39d5e4538071f0082b92768a854e6d43e47a708c932b6769e9: Status 404 returned error can't find the container with id b82dfa6b88731e39d5e4538071f0082b92768a854e6d43e47a708c932b6769e9
Jul 17 12:54:56.699129 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:56.699036    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6br26" event={"ID":"c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae","Type":"ContainerStarted","Data":"7b81ad231da0a72df1eb64e93ce17976a94227052b49823d8532c9b40e299cc3"}
Jul 17 12:54:56.699129 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:56.699088    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6br26" event={"ID":"c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae","Type":"ContainerStarted","Data":"b82dfa6b88731e39d5e4538071f0082b92768a854e6d43e47a708c932b6769e9"}
Jul 17 12:54:58.709074 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:58.709039    2564 generic.go:356] "Generic (PLEG): container finished" podID="c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae" containerID="7b81ad231da0a72df1eb64e93ce17976a94227052b49823d8532c9b40e299cc3" exitCode=0
Jul 17 12:54:58.709459 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:58.709115    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6br26" event={"ID":"c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae","Type":"ContainerDied","Data":"7b81ad231da0a72df1eb64e93ce17976a94227052b49823d8532c9b40e299cc3"}
Jul 17 12:54:59.714794 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:54:59.714760    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6br26" event={"ID":"c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae","Type":"ContainerStarted","Data":"2f081ea91b8bd2f184d32f1b0c9f803a86192d5038fb300f23514e68253e3ef9"}
Jul 17 12:55:05.746191 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:05.746153    2564 generic.go:356] "Generic (PLEG): container finished" podID="c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae" containerID="2f081ea91b8bd2f184d32f1b0c9f803a86192d5038fb300f23514e68253e3ef9" exitCode=0
Jul 17 12:55:05.746589 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:05.746227    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6br26" event={"ID":"c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae","Type":"ContainerDied","Data":"2f081ea91b8bd2f184d32f1b0c9f803a86192d5038fb300f23514e68253e3ef9"}
Jul 17 12:55:06.899896 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:06.899869    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6br26"
Jul 17 12:55:07.025713 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:07.025612    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gcgg\" (UniqueName: \"kubernetes.io/projected/c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae-kube-api-access-6gcgg\") pod \"c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae\" (UID: \"c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae\") "
Jul 17 12:55:07.025713 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:07.025683    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae-model-cache\") pod \"c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae\" (UID: \"c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae\") "
Jul 17 12:55:07.027673 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:07.027637    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae-kube-api-access-6gcgg" (OuterVolumeSpecName: "kube-api-access-6gcgg") pod "c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae" (UID: "c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae"). InnerVolumeSpecName "kube-api-access-6gcgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:55:07.047664 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:07.047629    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae-model-cache" (OuterVolumeSpecName: "model-cache") pod "c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae" (UID: "c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:55:07.126566 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:07.126531    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae-model-cache\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:55:07.126566 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:07.126562    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gcgg\" (UniqueName: \"kubernetes.io/projected/c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae-kube-api-access-6gcgg\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:55:07.760629 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:07.760605    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6br26"
Jul 17 12:55:07.760629 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:07.760612    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6br26" event={"ID":"c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae","Type":"ContainerDied","Data":"b82dfa6b88731e39d5e4538071f0082b92768a854e6d43e47a708c932b6769e9"}
Jul 17 12:55:07.760875 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:07.760656    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b82dfa6b88731e39d5e4538071f0082b92768a854e6d43e47a708c932b6769e9"
Jul 17 12:55:08.407038 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:08.407008    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:55:08.415774 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:08.415745    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-9cj2q"]
Jul 17 12:55:08.416101 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:08.416085    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae" containerName="download-hf-model"
Jul 17 12:55:08.416173 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:08.416104    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae" containerName="download-hf-model"
Jul 17 12:55:08.416173 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:08.416120    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae" containerName="s3-tls-init-custom"
Jul 17 12:55:08.416173 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:08.416129    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae" containerName="s3-tls-init-custom"
Jul 17 12:55:08.416314 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:08.416203    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae" containerName="s3-tls-init-custom"
Jul 17 12:55:08.420230 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:08.420210    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-9cj2q"
Jul 17 12:55:08.422947 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:08.422928    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 17 12:55:08.422947 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:08.422935    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Jul 17 12:55:08.429418 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:08.429393    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-9cj2q"]
Jul 17 12:55:08.529820 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:08.529784    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:55:08.538841 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:08.538802    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3c6c3bed-ad1b-48b4-b004-cbc7258e88a2-data\") pod \"seaweedfs-tls-serving-67b7477c6d-9cj2q\" (UID: \"3c6c3bed-ad1b-48b4-b004-cbc7258e88a2\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-9cj2q"
Jul 17 12:55:08.538960 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:08.538853    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x892n\" (UniqueName: \"kubernetes.io/projected/3c6c3bed-ad1b-48b4-b004-cbc7258e88a2-kube-api-access-x892n\") pod \"seaweedfs-tls-serving-67b7477c6d-9cj2q\" (UID: \"3c6c3bed-ad1b-48b4-b004-cbc7258e88a2\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-9cj2q"
Jul 17 12:55:08.538960 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:08.538876    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/3c6c3bed-ad1b-48b4-b004-cbc7258e88a2-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-9cj2q\" (UID: \"3c6c3bed-ad1b-48b4-b004-cbc7258e88a2\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-9cj2q"
Jul 17 12:55:08.639788 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:08.639747    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3c6c3bed-ad1b-48b4-b004-cbc7258e88a2-data\") pod \"seaweedfs-tls-serving-67b7477c6d-9cj2q\" (UID: \"3c6c3bed-ad1b-48b4-b004-cbc7258e88a2\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-9cj2q"
Jul 17 12:55:08.639788 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:08.639789    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x892n\" (UniqueName: \"kubernetes.io/projected/3c6c3bed-ad1b-48b4-b004-cbc7258e88a2-kube-api-access-x892n\") pod \"seaweedfs-tls-serving-67b7477c6d-9cj2q\" (UID: \"3c6c3bed-ad1b-48b4-b004-cbc7258e88a2\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-9cj2q"
Jul 17 12:55:08.639974 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:08.639810    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/3c6c3bed-ad1b-48b4-b004-cbc7258e88a2-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-9cj2q\" (UID: \"3c6c3bed-ad1b-48b4-b004-cbc7258e88a2\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-9cj2q"
Jul 17 12:55:08.640212 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:08.640193    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3c6c3bed-ad1b-48b4-b004-cbc7258e88a2-data\") pod \"seaweedfs-tls-serving-67b7477c6d-9cj2q\" (UID: \"3c6c3bed-ad1b-48b4-b004-cbc7258e88a2\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-9cj2q"
Jul 17 12:55:08.642320 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:08.642296    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/3c6c3bed-ad1b-48b4-b004-cbc7258e88a2-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-9cj2q\" (UID: \"3c6c3bed-ad1b-48b4-b004-cbc7258e88a2\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-9cj2q"
Jul 17 12:55:08.648370 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:08.648351    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x892n\" (UniqueName: \"kubernetes.io/projected/3c6c3bed-ad1b-48b4-b004-cbc7258e88a2-kube-api-access-x892n\") pod \"seaweedfs-tls-serving-67b7477c6d-9cj2q\" (UID: \"3c6c3bed-ad1b-48b4-b004-cbc7258e88a2\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-9cj2q"
Jul 17 12:55:08.730630 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:08.730531    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-9cj2q"
Jul 17 12:55:08.866061 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:08.866035    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-9cj2q"]
Jul 17 12:55:08.868265 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:55:08.868227    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c6c3bed_ad1b_48b4_b004_cbc7258e88a2.slice/crio-63c12ea5eda84c9d2fc751991fd038bd7ac882d7a78a2d59df766c37c840ba9b WatchSource:0}: Error finding container 63c12ea5eda84c9d2fc751991fd038bd7ac882d7a78a2d59df766c37c840ba9b: Status 404 returned error can't find the container with id 63c12ea5eda84c9d2fc751991fd038bd7ac882d7a78a2d59df766c37c840ba9b
Jul 17 12:55:09.770231 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:09.770192    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-9cj2q" event={"ID":"3c6c3bed-ad1b-48b4-b004-cbc7258e88a2","Type":"ContainerStarted","Data":"ecebd27448fda5df1b6c0cb310d861bdffd5975623d332f1c3bee094cd17968b"}
Jul 17 12:55:09.770787 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:09.770244    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-9cj2q" event={"ID":"3c6c3bed-ad1b-48b4-b004-cbc7258e88a2","Type":"ContainerStarted","Data":"63c12ea5eda84c9d2fc751991fd038bd7ac882d7a78a2d59df766c37c840ba9b"}
Jul 17 12:55:09.789128 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:09.789072    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-9cj2q" podStartSLOduration=1.5089296970000001 podStartE2EDuration="1.789053575s" totalImagesPullingTime="280.123878ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:55:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:55:09.788317701 +0000 UTC m=+1519.368429754" watchObservedRunningTime="2026-07-17 12:55:09.789053575 +0000 UTC m=+1519.369165600"
Jul 17 12:55:10.328691 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:10.328655    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-wvhnf"]
Jul 17 12:55:10.333209 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:10.333193    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-wvhnf"
Jul 17 12:55:10.341364 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:10.341332    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-wvhnf"]
Jul 17 12:55:10.454777 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:10.454743    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kss9\" (UniqueName: \"kubernetes.io/projected/a7aa9fd2-1266-444b-baec-91290b7ea5d7-kube-api-access-7kss9\") pod \"s3-tls-init-serving-wvhnf\" (UID: \"a7aa9fd2-1266-444b-baec-91290b7ea5d7\") " pod="kserve/s3-tls-init-serving-wvhnf"
Jul 17 12:55:10.454926 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:10.454800    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7aa9fd2-1266-444b-baec-91290b7ea5d7-model-cache\") pod \"s3-tls-init-serving-wvhnf\" (UID: \"a7aa9fd2-1266-444b-baec-91290b7ea5d7\") " pod="kserve/s3-tls-init-serving-wvhnf"
Jul 17 12:55:10.556000 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:10.555960    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kss9\" (UniqueName: \"kubernetes.io/projected/a7aa9fd2-1266-444b-baec-91290b7ea5d7-kube-api-access-7kss9\") pod \"s3-tls-init-serving-wvhnf\" (UID: \"a7aa9fd2-1266-444b-baec-91290b7ea5d7\") " pod="kserve/s3-tls-init-serving-wvhnf"
Jul 17 12:55:10.556000 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:10.556014    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7aa9fd2-1266-444b-baec-91290b7ea5d7-model-cache\") pod \"s3-tls-init-serving-wvhnf\" (UID: \"a7aa9fd2-1266-444b-baec-91290b7ea5d7\") " pod="kserve/s3-tls-init-serving-wvhnf"
Jul 17 12:55:10.556381 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:10.556363    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7aa9fd2-1266-444b-baec-91290b7ea5d7-model-cache\") pod \"s3-tls-init-serving-wvhnf\" (UID: \"a7aa9fd2-1266-444b-baec-91290b7ea5d7\") " pod="kserve/s3-tls-init-serving-wvhnf"
Jul 17 12:55:10.566042 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:10.566015    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kss9\" (UniqueName: \"kubernetes.io/projected/a7aa9fd2-1266-444b-baec-91290b7ea5d7-kube-api-access-7kss9\") pod \"s3-tls-init-serving-wvhnf\" (UID: \"a7aa9fd2-1266-444b-baec-91290b7ea5d7\") " pod="kserve/s3-tls-init-serving-wvhnf"
Jul 17 12:55:10.653973 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:10.653882    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-wvhnf"
Jul 17 12:55:10.788163 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:10.788129    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-wvhnf"]
Jul 17 12:55:10.791027 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:55:10.790998    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7aa9fd2_1266_444b_baec_91290b7ea5d7.slice/crio-87890a14748f028a8d000a60f86c8fc20a946595e5509ecaefc97f577f41affc WatchSource:0}: Error finding container 87890a14748f028a8d000a60f86c8fc20a946595e5509ecaefc97f577f41affc: Status 404 returned error can't find the container with id 87890a14748f028a8d000a60f86c8fc20a946595e5509ecaefc97f577f41affc
Jul 17 12:55:11.781339 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:11.781299    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-wvhnf" event={"ID":"a7aa9fd2-1266-444b-baec-91290b7ea5d7","Type":"ContainerStarted","Data":"80bff6b2262e30e866c9b849f06c985d89aeb9ab952dbb88259e6b0a8c2bcae0"}
Jul 17 12:55:11.781339 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:11.781345    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-wvhnf" event={"ID":"a7aa9fd2-1266-444b-baec-91290b7ea5d7","Type":"ContainerStarted","Data":"87890a14748f028a8d000a60f86c8fc20a946595e5509ecaefc97f577f41affc"}
Jul 17 12:55:12.786286 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:12.786203    2564 generic.go:356] "Generic (PLEG): container finished" podID="a7aa9fd2-1266-444b-baec-91290b7ea5d7" containerID="80bff6b2262e30e866c9b849f06c985d89aeb9ab952dbb88259e6b0a8c2bcae0" exitCode=0
Jul 17 12:55:12.786286 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:12.786242    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-wvhnf" event={"ID":"a7aa9fd2-1266-444b-baec-91290b7ea5d7","Type":"ContainerDied","Data":"80bff6b2262e30e866c9b849f06c985d89aeb9ab952dbb88259e6b0a8c2bcae0"}
Jul 17 12:55:13.792032 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:13.791985    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-wvhnf" event={"ID":"a7aa9fd2-1266-444b-baec-91290b7ea5d7","Type":"ContainerStarted","Data":"a43c8ecf0aae906e05be9801d9410d41da6a5f912e6edd4719034bb830323dd8"}
Jul 17 12:55:13.827819 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:13.827767    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-wvhnf" podStartSLOduration=2.653623924 podStartE2EDuration="3.827748367s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.174124443s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:55:13.824885384 +0000 UTC m=+1523.404997403" watchObservedRunningTime="2026-07-17 12:55:13.827748367 +0000 UTC m=+1523.407860385"
Jul 17 12:55:19.819076 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:19.819043    2564 generic.go:356] "Generic (PLEG): container finished" podID="a7aa9fd2-1266-444b-baec-91290b7ea5d7" containerID="a43c8ecf0aae906e05be9801d9410d41da6a5f912e6edd4719034bb830323dd8" exitCode=0
Jul 17 12:55:19.819574 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:19.819096    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-wvhnf" event={"ID":"a7aa9fd2-1266-444b-baec-91290b7ea5d7","Type":"ContainerDied","Data":"a43c8ecf0aae906e05be9801d9410d41da6a5f912e6edd4719034bb830323dd8"}
Jul 17 12:55:20.950484 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:20.950460    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-wvhnf"
Jul 17 12:55:21.041546 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:21.041511    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kss9\" (UniqueName: \"kubernetes.io/projected/a7aa9fd2-1266-444b-baec-91290b7ea5d7-kube-api-access-7kss9\") pod \"a7aa9fd2-1266-444b-baec-91290b7ea5d7\" (UID: \"a7aa9fd2-1266-444b-baec-91290b7ea5d7\") "
Jul 17 12:55:21.041761 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:21.041594    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7aa9fd2-1266-444b-baec-91290b7ea5d7-model-cache\") pod \"a7aa9fd2-1266-444b-baec-91290b7ea5d7\" (UID: \"a7aa9fd2-1266-444b-baec-91290b7ea5d7\") "
Jul 17 12:55:21.043600 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:21.043567    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7aa9fd2-1266-444b-baec-91290b7ea5d7-kube-api-access-7kss9" (OuterVolumeSpecName: "kube-api-access-7kss9") pod "a7aa9fd2-1266-444b-baec-91290b7ea5d7" (UID: "a7aa9fd2-1266-444b-baec-91290b7ea5d7"). InnerVolumeSpecName "kube-api-access-7kss9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:55:21.063020 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:21.062969    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7aa9fd2-1266-444b-baec-91290b7ea5d7-model-cache" (OuterVolumeSpecName: "model-cache") pod "a7aa9fd2-1266-444b-baec-91290b7ea5d7" (UID: "a7aa9fd2-1266-444b-baec-91290b7ea5d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:55:21.142851 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:21.142785    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7aa9fd2-1266-444b-baec-91290b7ea5d7-model-cache\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:55:21.142851 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:21.142808    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kss9\" (UniqueName: \"kubernetes.io/projected/a7aa9fd2-1266-444b-baec-91290b7ea5d7-kube-api-access-7kss9\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:55:21.828226 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:21.828191    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-wvhnf"
Jul 17 12:55:21.828226 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:21.828212    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-wvhnf" event={"ID":"a7aa9fd2-1266-444b-baec-91290b7ea5d7","Type":"ContainerDied","Data":"87890a14748f028a8d000a60f86c8fc20a946595e5509ecaefc97f577f41affc"}
Jul 17 12:55:21.828429 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:21.828247    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87890a14748f028a8d000a60f86c8fc20a946595e5509ecaefc97f577f41affc"
Jul 17 12:55:22.187514 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:22.187418    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:55:22.396031 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:22.395997    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:55:33.814658 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:33.814619    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"]
Jul 17 12:55:33.815068 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:33.814950    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7aa9fd2-1266-444b-baec-91290b7ea5d7" containerName="download-hf-model"
Jul 17 12:55:33.815068 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:33.814961    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7aa9fd2-1266-444b-baec-91290b7ea5d7" containerName="download-hf-model"
Jul 17 12:55:33.815068 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:33.814979    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7aa9fd2-1266-444b-baec-91290b7ea5d7" containerName="s3-tls-init-serving"
Jul 17 12:55:33.815068 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:33.814984    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7aa9fd2-1266-444b-baec-91290b7ea5d7" containerName="s3-tls-init-serving"
Jul 17 12:55:33.815068 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:33.815027    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7aa9fd2-1266-444b-baec-91290b7ea5d7" containerName="s3-tls-init-serving"
Jul 17 12:55:33.818534 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:33.818515    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"
Jul 17 12:55:33.821718 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:33.821687    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 17 12:55:33.824512 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:33.824493    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 17 12:55:33.824649 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:33.824630    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7v8wk\""
Jul 17 12:55:33.824738 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:33.824647    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 17 12:55:33.824810 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:33.824737    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 17 12:55:33.824864 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:33.824826    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 17 12:55:33.825563 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:33.825494    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 17 12:55:33.844434 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:33.844400    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"]
Jul 17 12:55:33.956934 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:33.956886    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26c4e402-5a31-4b5f-9863-a71855ea1c2b-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9\" (UID: \"26c4e402-5a31-4b5f-9863-a71855ea1c2b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"
Jul 17 12:55:33.957113 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:33.956964    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26c4e402-5a31-4b5f-9863-a71855ea1c2b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9\" (UID: \"26c4e402-5a31-4b5f-9863-a71855ea1c2b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"
Jul 17 12:55:33.957113 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:33.956988    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br7pr\" (UniqueName: \"kubernetes.io/projected/26c4e402-5a31-4b5f-9863-a71855ea1c2b-kube-api-access-br7pr\") pod \"isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9\" (UID: \"26c4e402-5a31-4b5f-9863-a71855ea1c2b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"
Jul 17 12:55:33.957113 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:33.957007    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26c4e402-5a31-4b5f-9863-a71855ea1c2b-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9\" (UID: \"26c4e402-5a31-4b5f-9863-a71855ea1c2b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"
Jul 17 12:55:33.957113 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:33.957084    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26c4e402-5a31-4b5f-9863-a71855ea1c2b-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9\" (UID: \"26c4e402-5a31-4b5f-9863-a71855ea1c2b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"
Jul 17 12:55:34.058553 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:34.058512    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br7pr\" (UniqueName: \"kubernetes.io/projected/26c4e402-5a31-4b5f-9863-a71855ea1c2b-kube-api-access-br7pr\") pod \"isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9\" (UID: \"26c4e402-5a31-4b5f-9863-a71855ea1c2b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"
Jul 17 12:55:34.058754 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:34.058571    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26c4e402-5a31-4b5f-9863-a71855ea1c2b-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9\" (UID: \"26c4e402-5a31-4b5f-9863-a71855ea1c2b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"
Jul 17 12:55:34.058754 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:34.058614    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26c4e402-5a31-4b5f-9863-a71855ea1c2b-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9\" (UID: \"26c4e402-5a31-4b5f-9863-a71855ea1c2b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"
Jul 17 12:55:34.058754 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:34.058644    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26c4e402-5a31-4b5f-9863-a71855ea1c2b-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9\" (UID: \"26c4e402-5a31-4b5f-9863-a71855ea1c2b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"
Jul 17 12:55:34.058942 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:34.058754    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26c4e402-5a31-4b5f-9863-a71855ea1c2b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9\" (UID: \"26c4e402-5a31-4b5f-9863-a71855ea1c2b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"
Jul 17 12:55:34.059136 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:34.059110    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26c4e402-5a31-4b5f-9863-a71855ea1c2b-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9\" (UID: \"26c4e402-5a31-4b5f-9863-a71855ea1c2b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"
Jul 17 12:55:34.059439 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:34.059416    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26c4e402-5a31-4b5f-9863-a71855ea1c2b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9\" (UID: \"26c4e402-5a31-4b5f-9863-a71855ea1c2b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"
Jul 17 12:55:34.059490 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:34.059425    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26c4e402-5a31-4b5f-9863-a71855ea1c2b-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9\" (UID: \"26c4e402-5a31-4b5f-9863-a71855ea1c2b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"
Jul 17 12:55:34.061272 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:34.061253    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26c4e402-5a31-4b5f-9863-a71855ea1c2b-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9\" (UID: \"26c4e402-5a31-4b5f-9863-a71855ea1c2b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"
Jul 17 12:55:34.071955 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:34.071892    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-br7pr\" (UniqueName: \"kubernetes.io/projected/26c4e402-5a31-4b5f-9863-a71855ea1c2b-kube-api-access-br7pr\") pod \"isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9\" (UID: \"26c4e402-5a31-4b5f-9863-a71855ea1c2b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"
Jul 17 12:55:34.129473 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:34.129431    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"
Jul 17 12:55:34.264862 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:34.264839    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"]
Jul 17 12:55:34.266587 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:55:34.266563    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c4e402_5a31_4b5f_9863_a71855ea1c2b.slice/crio-71731e0cf9eec4d74be388091a11e9b14074399bda0999b6f7ca2a764171e79f WatchSource:0}: Error finding container 71731e0cf9eec4d74be388091a11e9b14074399bda0999b6f7ca2a764171e79f: Status 404 returned error can't find the container with id 71731e0cf9eec4d74be388091a11e9b14074399bda0999b6f7ca2a764171e79f
Jul 17 12:55:34.797903 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:34.797864    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:55:34.882948 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:34.882911    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" event={"ID":"26c4e402-5a31-4b5f-9863-a71855ea1c2b","Type":"ContainerStarted","Data":"960f321a10e76f0d1d73c424c54ed705e43059253f08c553cdede5d15c101ac0"}
Jul 17 12:55:34.883363 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:34.882988    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" event={"ID":"26c4e402-5a31-4b5f-9863-a71855ea1c2b","Type":"ContainerStarted","Data":"71731e0cf9eec4d74be388091a11e9b14074399bda0999b6f7ca2a764171e79f"}
Jul 17 12:55:34.909538 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:34.909502    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:55:37.895926 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:37.895887    2564 generic.go:356] "Generic (PLEG): container finished" podID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerID="960f321a10e76f0d1d73c424c54ed705e43059253f08c553cdede5d15c101ac0" exitCode=0
Jul 17 12:55:37.896345 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:37.895963    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" event={"ID":"26c4e402-5a31-4b5f-9863-a71855ea1c2b","Type":"ContainerDied","Data":"960f321a10e76f0d1d73c424c54ed705e43059253f08c553cdede5d15c101ac0"}
Jul 17 12:55:51.974169 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:51.974090    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" event={"ID":"26c4e402-5a31-4b5f-9863-a71855ea1c2b","Type":"ContainerStarted","Data":"40684c1c07583332fd9574808bd5dcae2d88301cffb8dedd8cbefff57fb25e91"}
Jul 17 12:55:54.992299 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:54.992189    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" event={"ID":"26c4e402-5a31-4b5f-9863-a71855ea1c2b","Type":"ContainerStarted","Data":"f36ab743edcfcc610025da165d9ecff160faab54f41e3fee92ad5a8f7298d123"}
Jul 17 12:55:57.003951 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:57.003915    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" event={"ID":"26c4e402-5a31-4b5f-9863-a71855ea1c2b","Type":"ContainerStarted","Data":"ac45823be4c3cc5988107c7e150fe542e324cb5cf2e71fadc1e119f5c72fb037"}
Jul 17 12:55:57.004387 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:57.004113    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"
Jul 17 12:55:57.004387 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:57.004283    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"
Jul 17 12:55:57.005878 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:57.005851    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 17 12:55:57.030673 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:57.030621    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podStartSLOduration=1.651103987 podStartE2EDuration="24.030602604s" totalImagesPullingTime="18.889982634s" totalInitContainerRuntime="3.489515983s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:55:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:55:57.02931051 +0000 UTC m=+1566.609422522" watchObservedRunningTime="2026-07-17 12:55:57.030602604 +0000 UTC m=+1566.610714623"
Jul 17 12:55:58.008303 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:58.008261    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"
Jul 17 12:55:58.008303 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:58.008282    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 17 12:55:58.009509 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:58.009485    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:55:59.012372 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:59.012331    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 17 12:55:59.012863 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:59.012828    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:55:59.016390 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:55:59.016369    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"
Jul 17 12:56:00.016971 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:56:00.016930    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 17 12:56:00.017356 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:56:00.017259    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:56:10.017520 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:56:10.017465    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 17 12:56:10.017974 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:56:10.017866    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:56:20.016961 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:56:20.016909    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 17 12:56:20.017388 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:56:20.017363    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:56:30.017523 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:56:30.017457    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 17 12:56:30.018174 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:56:30.017936    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:56:40.016978 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:56:40.016931    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 17 12:56:40.017410 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:56:40.017381    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:56:50.017395 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:56:50.017304    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 17 12:56:50.017852 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:56:50.017769    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:57:00.016967 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:00.016916    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 17 12:57:00.017425 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:00.017399    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:57:05.912962 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:05.912920    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:57:05.967820 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:05.967775    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:57:10.018590 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:10.018555    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"
Jul 17 12:57:10.019074 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:10.018684    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"
Jul 17 12:57:18.812056 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:18.812023    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"]
Jul 17 12:57:18.812616 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:18.812271    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="kserve-container" containerID="cri-o://40684c1c07583332fd9574808bd5dcae2d88301cffb8dedd8cbefff57fb25e91" gracePeriod=30
Jul 17 12:57:18.812616 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:18.812323    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="agent" containerID="cri-o://ac45823be4c3cc5988107c7e150fe542e324cb5cf2e71fadc1e119f5c72fb037" gracePeriod=30
Jul 17 12:57:18.812616 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:18.812325    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="kube-rbac-proxy" containerID="cri-o://f36ab743edcfcc610025da165d9ecff160faab54f41e3fee92ad5a8f7298d123" gracePeriod=30
Jul 17 12:57:18.838861 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:18.838831    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:57:18.922691 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:18.922663    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:57:18.927227 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:18.927197    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"]
Jul 17 12:57:18.930851 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:18.930832    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"
Jul 17 12:57:18.933517 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:18.933462    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 17 12:57:18.933640 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:18.933568    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 17 12:57:18.943912 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:18.943886    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"]
Jul 17 12:57:19.013195 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:19.013146    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 17 12:57:19.099737 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:19.099619    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0e9ad9e-af60-4f29-88f1-96dda14d647e-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn\" (UID: \"e0e9ad9e-af60-4f29-88f1-96dda14d647e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"
Jul 17 12:57:19.099737 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:19.099665    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0e9ad9e-af60-4f29-88f1-96dda14d647e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn\" (UID: \"e0e9ad9e-af60-4f29-88f1-96dda14d647e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"
Jul 17 12:57:19.099925 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:19.099732    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0e9ad9e-af60-4f29-88f1-96dda14d647e-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn\" (UID: \"e0e9ad9e-af60-4f29-88f1-96dda14d647e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"
Jul 17 12:57:19.099925 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:19.099786    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5pdb\" (UniqueName: \"kubernetes.io/projected/e0e9ad9e-af60-4f29-88f1-96dda14d647e-kube-api-access-l5pdb\") pod \"isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn\" (UID: \"e0e9ad9e-af60-4f29-88f1-96dda14d647e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"
Jul 17 12:57:19.099925 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:19.099808    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0e9ad9e-af60-4f29-88f1-96dda14d647e-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn\" (UID: \"e0e9ad9e-af60-4f29-88f1-96dda14d647e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"
Jul 17 12:57:19.165215 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:19.165172    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:57:19.200594 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:19.200555    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0e9ad9e-af60-4f29-88f1-96dda14d647e-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn\" (UID: \"e0e9ad9e-af60-4f29-88f1-96dda14d647e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"
Jul 17 12:57:19.200830 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:19.200610    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0e9ad9e-af60-4f29-88f1-96dda14d647e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn\" (UID: \"e0e9ad9e-af60-4f29-88f1-96dda14d647e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"
Jul 17 12:57:19.200830 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:19.200651    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0e9ad9e-af60-4f29-88f1-96dda14d647e-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn\" (UID: \"e0e9ad9e-af60-4f29-88f1-96dda14d647e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"
Jul 17 12:57:19.200830 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:19.200723    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5pdb\" (UniqueName: \"kubernetes.io/projected/e0e9ad9e-af60-4f29-88f1-96dda14d647e-kube-api-access-l5pdb\") pod \"isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn\" (UID: \"e0e9ad9e-af60-4f29-88f1-96dda14d647e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"
Jul 17 12:57:19.200830 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:19.200747    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0e9ad9e-af60-4f29-88f1-96dda14d647e-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn\" (UID: \"e0e9ad9e-af60-4f29-88f1-96dda14d647e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"
Jul 17 12:57:19.201194 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:19.201174    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0e9ad9e-af60-4f29-88f1-96dda14d647e-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn\" (UID: \"e0e9ad9e-af60-4f29-88f1-96dda14d647e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"
Jul 17 12:57:19.201433 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:19.201413    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0e9ad9e-af60-4f29-88f1-96dda14d647e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn\" (UID: \"e0e9ad9e-af60-4f29-88f1-96dda14d647e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"
Jul 17 12:57:19.201513 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:19.201432    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0e9ad9e-af60-4f29-88f1-96dda14d647e-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn\" (UID: \"e0e9ad9e-af60-4f29-88f1-96dda14d647e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"
Jul 17 12:57:19.203133 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:19.203115    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0e9ad9e-af60-4f29-88f1-96dda14d647e-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn\" (UID: \"e0e9ad9e-af60-4f29-88f1-96dda14d647e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"
Jul 17 12:57:19.210380 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:19.210357    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5pdb\" (UniqueName: \"kubernetes.io/projected/e0e9ad9e-af60-4f29-88f1-96dda14d647e-kube-api-access-l5pdb\") pod \"isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn\" (UID: \"e0e9ad9e-af60-4f29-88f1-96dda14d647e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"
Jul 17 12:57:19.242072 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:19.242012    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"
Jul 17 12:57:19.346065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:19.345886    2564 generic.go:356] "Generic (PLEG): container finished" podID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerID="f36ab743edcfcc610025da165d9ecff160faab54f41e3fee92ad5a8f7298d123" exitCode=2
Jul 17 12:57:19.346065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:19.345935    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" event={"ID":"26c4e402-5a31-4b5f-9863-a71855ea1c2b","Type":"ContainerDied","Data":"f36ab743edcfcc610025da165d9ecff160faab54f41e3fee92ad5a8f7298d123"}
Jul 17 12:57:19.378159 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:19.378124    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"]
Jul 17 12:57:19.380640 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:57:19.380614    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e9ad9e_af60_4f29_88f1_96dda14d647e.slice/crio-99acd60ad2c43aa1c7c1507848dcd1342a9b91cc55b1bb32dea1bb19125c6f81 WatchSource:0}: Error finding container 99acd60ad2c43aa1c7c1507848dcd1342a9b91cc55b1bb32dea1bb19125c6f81: Status 404 returned error can't find the container with id 99acd60ad2c43aa1c7c1507848dcd1342a9b91cc55b1bb32dea1bb19125c6f81
Jul 17 12:57:20.017906 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:20.017862    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 17 12:57:20.018333 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:20.018255    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:57:20.351977 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:20.351934    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" event={"ID":"e0e9ad9e-af60-4f29-88f1-96dda14d647e","Type":"ContainerStarted","Data":"d8a3bbce2a3801aba179d066b73c59d3b1c9148e15d35e585af8243cbba1367e"}
Jul 17 12:57:20.352145 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:20.352005    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" event={"ID":"e0e9ad9e-af60-4f29-88f1-96dda14d647e","Type":"ContainerStarted","Data":"99acd60ad2c43aa1c7c1507848dcd1342a9b91cc55b1bb32dea1bb19125c6f81"}
Jul 17 12:57:23.366498 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:23.366465    2564 generic.go:356] "Generic (PLEG): container finished" podID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerID="d8a3bbce2a3801aba179d066b73c59d3b1c9148e15d35e585af8243cbba1367e" exitCode=0
Jul 17 12:57:23.366921 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:23.366541    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" event={"ID":"e0e9ad9e-af60-4f29-88f1-96dda14d647e","Type":"ContainerDied","Data":"d8a3bbce2a3801aba179d066b73c59d3b1c9148e15d35e585af8243cbba1367e"}
Jul 17 12:57:24.013628 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:24.013530    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 17 12:57:24.373067 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:24.373007    2564 generic.go:356] "Generic (PLEG): container finished" podID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerID="40684c1c07583332fd9574808bd5dcae2d88301cffb8dedd8cbefff57fb25e91" exitCode=0
Jul 17 12:57:24.373553 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:24.373089    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" event={"ID":"26c4e402-5a31-4b5f-9863-a71855ea1c2b","Type":"ContainerDied","Data":"40684c1c07583332fd9574808bd5dcae2d88301cffb8dedd8cbefff57fb25e91"}
Jul 17 12:57:24.375572 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:24.375546    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" event={"ID":"e0e9ad9e-af60-4f29-88f1-96dda14d647e","Type":"ContainerStarted","Data":"dbeecc078d40133bde3b11e10080d945febbf9239749a542307fe6b8ab1acbd1"}
Jul 17 12:57:24.375690 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:24.375605    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" event={"ID":"e0e9ad9e-af60-4f29-88f1-96dda14d647e","Type":"ContainerStarted","Data":"d06d847825146a58083fe6cefed1e9f7c03a9c9867b0822969dc6797a15160d3"}
Jul 17 12:57:24.375690 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:24.375636    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" event={"ID":"e0e9ad9e-af60-4f29-88f1-96dda14d647e","Type":"ContainerStarted","Data":"275f0527abd7bbac01a174f6de479d7778ea520e6c09fd6db7667279c6bd489a"}
Jul 17 12:57:24.375953 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:24.375934    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"
Jul 17 12:57:24.376065 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:24.376046    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"
Jul 17 12:57:24.377228 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:24.377205    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused"
Jul 17 12:57:24.397292 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:24.397243    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podStartSLOduration=2.651980535 podStartE2EDuration="6.397228086s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.745247551s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:57:24.394860275 +0000 UTC m=+1653.974972352" watchObservedRunningTime="2026-07-17 12:57:24.397228086 +0000 UTC m=+1653.977340137"
Jul 17 12:57:25.380281 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:25.380248    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"
Jul 17 12:57:25.380713 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:25.380422    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused"
Jul 17 12:57:25.381355 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:25.381329    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:57:26.384988 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:26.384944    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused"
Jul 17 12:57:26.385486 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:26.385279    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:57:29.012951 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:29.012912    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 17 12:57:29.013651 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:29.013633    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"
Jul 17 12:57:30.017092 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:30.017037    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 17 12:57:30.017570 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:30.017394    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:57:31.389209 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:31.389178    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"
Jul 17 12:57:31.389892 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:31.389855    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused"
Jul 17 12:57:31.390158 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:31.390128    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:57:34.013588 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:34.013544    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 17 12:57:39.013113 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:39.013070    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 17 12:57:40.017173 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:40.017127    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 17 12:57:40.017622 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:40.017275    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"
Jul 17 12:57:40.017622 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:40.017457    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:57:40.017622 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:40.017551    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"
Jul 17 12:57:41.390076 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:41.390039    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused"
Jul 17 12:57:41.390526 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:41.390410    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:57:44.013191 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:44.013149    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 17 12:57:48.961992 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:48.961964    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"
Jul 17 12:57:49.062878 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.062835    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26c4e402-5a31-4b5f-9863-a71855ea1c2b-kube-rbac-proxy-sar-config\") pod \"26c4e402-5a31-4b5f-9863-a71855ea1c2b\" (UID: \"26c4e402-5a31-4b5f-9863-a71855ea1c2b\") "
Jul 17 12:57:49.062878 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.062872    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-br7pr\" (UniqueName: \"kubernetes.io/projected/26c4e402-5a31-4b5f-9863-a71855ea1c2b-kube-api-access-br7pr\") pod \"26c4e402-5a31-4b5f-9863-a71855ea1c2b\" (UID: \"26c4e402-5a31-4b5f-9863-a71855ea1c2b\") "
Jul 17 12:57:49.063160 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.062921    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26c4e402-5a31-4b5f-9863-a71855ea1c2b-kserve-provision-location\") pod \"26c4e402-5a31-4b5f-9863-a71855ea1c2b\" (UID: \"26c4e402-5a31-4b5f-9863-a71855ea1c2b\") "
Jul 17 12:57:49.063160 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.062936    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26c4e402-5a31-4b5f-9863-a71855ea1c2b-cabundle-cert\") pod \"26c4e402-5a31-4b5f-9863-a71855ea1c2b\" (UID: \"26c4e402-5a31-4b5f-9863-a71855ea1c2b\") "
Jul 17 12:57:49.063160 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.062980    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26c4e402-5a31-4b5f-9863-a71855ea1c2b-proxy-tls\") pod \"26c4e402-5a31-4b5f-9863-a71855ea1c2b\" (UID: \"26c4e402-5a31-4b5f-9863-a71855ea1c2b\") "
Jul 17 12:57:49.063397 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.063281    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c4e402-5a31-4b5f-9863-a71855ea1c2b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "26c4e402-5a31-4b5f-9863-a71855ea1c2b" (UID: "26c4e402-5a31-4b5f-9863-a71855ea1c2b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:57:49.063529 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.063271    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c4e402-5a31-4b5f-9863-a71855ea1c2b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "26c4e402-5a31-4b5f-9863-a71855ea1c2b" (UID: "26c4e402-5a31-4b5f-9863-a71855ea1c2b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:57:49.063529 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.063447    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c4e402-5a31-4b5f-9863-a71855ea1c2b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "26c4e402-5a31-4b5f-9863-a71855ea1c2b" (UID: "26c4e402-5a31-4b5f-9863-a71855ea1c2b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:57:49.065269 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.065245    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c4e402-5a31-4b5f-9863-a71855ea1c2b-kube-api-access-br7pr" (OuterVolumeSpecName: "kube-api-access-br7pr") pod "26c4e402-5a31-4b5f-9863-a71855ea1c2b" (UID: "26c4e402-5a31-4b5f-9863-a71855ea1c2b"). InnerVolumeSpecName "kube-api-access-br7pr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:57:49.065370 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.065282    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26c4e402-5a31-4b5f-9863-a71855ea1c2b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "26c4e402-5a31-4b5f-9863-a71855ea1c2b" (UID: "26c4e402-5a31-4b5f-9863-a71855ea1c2b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:57:49.163632 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.163596    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26c4e402-5a31-4b5f-9863-a71855ea1c2b-proxy-tls\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:57:49.163632 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.163624    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26c4e402-5a31-4b5f-9863-a71855ea1c2b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:57:49.163632 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.163636    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br7pr\" (UniqueName: \"kubernetes.io/projected/26c4e402-5a31-4b5f-9863-a71855ea1c2b-kube-api-access-br7pr\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:57:49.163962 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.163645    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26c4e402-5a31-4b5f-9863-a71855ea1c2b-kserve-provision-location\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:57:49.163962 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.163656    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26c4e402-5a31-4b5f-9863-a71855ea1c2b-cabundle-cert\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:57:49.482054 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.481963    2564 generic.go:356] "Generic (PLEG): container finished" podID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerID="ac45823be4c3cc5988107c7e150fe542e324cb5cf2e71fadc1e119f5c72fb037" exitCode=0
Jul 17 12:57:49.482054 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.482016    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"
Jul 17 12:57:49.482254 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.482077    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" event={"ID":"26c4e402-5a31-4b5f-9863-a71855ea1c2b","Type":"ContainerDied","Data":"ac45823be4c3cc5988107c7e150fe542e324cb5cf2e71fadc1e119f5c72fb037"}
Jul 17 12:57:49.482254 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.482143    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9" event={"ID":"26c4e402-5a31-4b5f-9863-a71855ea1c2b","Type":"ContainerDied","Data":"71731e0cf9eec4d74be388091a11e9b14074399bda0999b6f7ca2a764171e79f"}
Jul 17 12:57:49.482254 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.482175    2564 scope.go:117] "RemoveContainer" containerID="ac45823be4c3cc5988107c7e150fe542e324cb5cf2e71fadc1e119f5c72fb037"
Jul 17 12:57:49.491026 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.491008    2564 scope.go:117] "RemoveContainer" containerID="f36ab743edcfcc610025da165d9ecff160faab54f41e3fee92ad5a8f7298d123"
Jul 17 12:57:49.498600 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.498581    2564 scope.go:117] "RemoveContainer" containerID="40684c1c07583332fd9574808bd5dcae2d88301cffb8dedd8cbefff57fb25e91"
Jul 17 12:57:49.506913 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.506865    2564 scope.go:117] "RemoveContainer" containerID="960f321a10e76f0d1d73c424c54ed705e43059253f08c553cdede5d15c101ac0"
Jul 17 12:57:49.507988 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.507969    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"]
Jul 17 12:57:49.513671 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.513648    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9d5b8f96-cdkl9"]
Jul 17 12:57:49.515173 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.515157    2564 scope.go:117] "RemoveContainer" containerID="ac45823be4c3cc5988107c7e150fe542e324cb5cf2e71fadc1e119f5c72fb037"
Jul 17 12:57:49.515453 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:57:49.515437    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac45823be4c3cc5988107c7e150fe542e324cb5cf2e71fadc1e119f5c72fb037\": container with ID starting with ac45823be4c3cc5988107c7e150fe542e324cb5cf2e71fadc1e119f5c72fb037 not found: ID does not exist" containerID="ac45823be4c3cc5988107c7e150fe542e324cb5cf2e71fadc1e119f5c72fb037"
Jul 17 12:57:49.515507 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.515461    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac45823be4c3cc5988107c7e150fe542e324cb5cf2e71fadc1e119f5c72fb037"} err="failed to get container status \"ac45823be4c3cc5988107c7e150fe542e324cb5cf2e71fadc1e119f5c72fb037\": rpc error: code = NotFound desc = could not find container \"ac45823be4c3cc5988107c7e150fe542e324cb5cf2e71fadc1e119f5c72fb037\": container with ID starting with ac45823be4c3cc5988107c7e150fe542e324cb5cf2e71fadc1e119f5c72fb037 not found: ID does not exist"
Jul 17 12:57:49.515507 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.515478    2564 scope.go:117] "RemoveContainer" containerID="f36ab743edcfcc610025da165d9ecff160faab54f41e3fee92ad5a8f7298d123"
Jul 17 12:57:49.515756 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:57:49.515738    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f36ab743edcfcc610025da165d9ecff160faab54f41e3fee92ad5a8f7298d123\": container with ID starting with f36ab743edcfcc610025da165d9ecff160faab54f41e3fee92ad5a8f7298d123 not found: ID does not exist" containerID="f36ab743edcfcc610025da165d9ecff160faab54f41e3fee92ad5a8f7298d123"
Jul 17 12:57:49.515831 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.515765    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f36ab743edcfcc610025da165d9ecff160faab54f41e3fee92ad5a8f7298d123"} err="failed to get container status \"f36ab743edcfcc610025da165d9ecff160faab54f41e3fee92ad5a8f7298d123\": rpc error: code = NotFound desc = could not find container \"f36ab743edcfcc610025da165d9ecff160faab54f41e3fee92ad5a8f7298d123\": container with ID starting with f36ab743edcfcc610025da165d9ecff160faab54f41e3fee92ad5a8f7298d123 not found: ID does not exist"
Jul 17 12:57:49.515831 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.515787    2564 scope.go:117] "RemoveContainer" containerID="40684c1c07583332fd9574808bd5dcae2d88301cffb8dedd8cbefff57fb25e91"
Jul 17 12:57:49.516063 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:57:49.516042    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40684c1c07583332fd9574808bd5dcae2d88301cffb8dedd8cbefff57fb25e91\": container with ID starting with 40684c1c07583332fd9574808bd5dcae2d88301cffb8dedd8cbefff57fb25e91 not found: ID does not exist" containerID="40684c1c07583332fd9574808bd5dcae2d88301cffb8dedd8cbefff57fb25e91"
Jul 17 12:57:49.516063 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.516064    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40684c1c07583332fd9574808bd5dcae2d88301cffb8dedd8cbefff57fb25e91"} err="failed to get container status \"40684c1c07583332fd9574808bd5dcae2d88301cffb8dedd8cbefff57fb25e91\": rpc error: code = NotFound desc = could not find container \"40684c1c07583332fd9574808bd5dcae2d88301cffb8dedd8cbefff57fb25e91\": container with ID starting with 40684c1c07583332fd9574808bd5dcae2d88301cffb8dedd8cbefff57fb25e91 not found: ID does not exist"
Jul 17 12:57:49.516197 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.516079    2564 scope.go:117] "RemoveContainer" containerID="960f321a10e76f0d1d73c424c54ed705e43059253f08c553cdede5d15c101ac0"
Jul 17 12:57:49.516323 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:57:49.516283    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"960f321a10e76f0d1d73c424c54ed705e43059253f08c553cdede5d15c101ac0\": container with ID starting with 960f321a10e76f0d1d73c424c54ed705e43059253f08c553cdede5d15c101ac0 not found: ID does not exist" containerID="960f321a10e76f0d1d73c424c54ed705e43059253f08c553cdede5d15c101ac0"
Jul 17 12:57:49.516400 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:49.516327    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"960f321a10e76f0d1d73c424c54ed705e43059253f08c553cdede5d15c101ac0"} err="failed to get container status \"960f321a10e76f0d1d73c424c54ed705e43059253f08c553cdede5d15c101ac0\": rpc error: code = NotFound desc = could not find container \"960f321a10e76f0d1d73c424c54ed705e43059253f08c553cdede5d15c101ac0\": container with ID starting with 960f321a10e76f0d1d73c424c54ed705e43059253f08c553cdede5d15c101ac0 not found: ID does not exist"
Jul 17 12:57:51.015507 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:51.015473    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" path="/var/lib/kubelet/pods/26c4e402-5a31-4b5f-9863-a71855ea1c2b/volumes"
Jul 17 12:57:51.390767 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:51.390727    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused"
Jul 17 12:57:51.391212 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:57:51.391187    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:58:01.389979 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:01.389930    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused"
Jul 17 12:58:01.390398 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:01.390370    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:58:11.390120 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:11.390074    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused"
Jul 17 12:58:11.392546 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:11.390472    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:58:21.390254 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:21.390201    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused"
Jul 17 12:58:21.390734 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:21.390642    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:58:31.389997 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:31.389951    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused"
Jul 17 12:58:31.390471 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:31.390178    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:58:41.390372 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:41.390338    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"
Jul 17 12:58:41.390873 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:41.390548    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"
Jul 17 12:58:54.100894 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.100852    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"]
Jul 17 12:58:54.101452 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.101260    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="kserve-container" containerID="cri-o://275f0527abd7bbac01a174f6de479d7778ea520e6c09fd6db7667279c6bd489a" gracePeriod=30
Jul 17 12:58:54.101452 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.101310    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="agent" containerID="cri-o://dbeecc078d40133bde3b11e10080d945febbf9239749a542307fe6b8ab1acbd1" gracePeriod=30
Jul 17 12:58:54.101452 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.101338    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="kube-rbac-proxy" containerID="cri-o://d06d847825146a58083fe6cefed1e9f7c03a9c9867b0822969dc6797a15160d3" gracePeriod=30
Jul 17 12:58:54.160040 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.159978    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:58:54.185595 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.185559    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj"]
Jul 17 12:58:54.185915 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.185902    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="kube-rbac-proxy"
Jul 17 12:58:54.185964 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.185917    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="kube-rbac-proxy"
Jul 17 12:58:54.185964 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.185931    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="agent"
Jul 17 12:58:54.185964 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.185937    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="agent"
Jul 17 12:58:54.185964 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.185954    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="storage-initializer"
Jul 17 12:58:54.185964 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.185959    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="storage-initializer"
Jul 17 12:58:54.186140 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.185967    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="kserve-container"
Jul 17 12:58:54.186140 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.185972    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="kserve-container"
Jul 17 12:58:54.186140 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.186048    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="kserve-container"
Jul 17 12:58:54.186140 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.186060    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="agent"
Jul 17 12:58:54.186140 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.186066    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="26c4e402-5a31-4b5f-9863-a71855ea1c2b" containerName="kube-rbac-proxy"
Jul 17 12:58:54.189250 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.189227    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj"
Jul 17 12:58:54.192918 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.192894    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 17 12:58:54.193058 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.193034    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 17 12:58:54.205729 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.205687    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj"]
Jul 17 12:58:54.309393 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.309359    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/752cd404-5062-457f-9223-152507394688-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-2c4kj\" (UID: \"752cd404-5062-457f-9223-152507394688\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj"
Jul 17 12:58:54.309393 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.309404    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrdbb\" (UniqueName: \"kubernetes.io/projected/752cd404-5062-457f-9223-152507394688-kube-api-access-nrdbb\") pod \"message-dumper-predictor-6575bcfdfd-2c4kj\" (UID: \"752cd404-5062-457f-9223-152507394688\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj"
Jul 17 12:58:54.309645 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.309486    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/752cd404-5062-457f-9223-152507394688-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-2c4kj\" (UID: \"752cd404-5062-457f-9223-152507394688\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj"
Jul 17 12:58:54.410402 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.410299    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/752cd404-5062-457f-9223-152507394688-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-2c4kj\" (UID: \"752cd404-5062-457f-9223-152507394688\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj"
Jul 17 12:58:54.410402 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.410349    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrdbb\" (UniqueName: \"kubernetes.io/projected/752cd404-5062-457f-9223-152507394688-kube-api-access-nrdbb\") pod \"message-dumper-predictor-6575bcfdfd-2c4kj\" (UID: \"752cd404-5062-457f-9223-152507394688\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj"
Jul 17 12:58:54.410402 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.410401    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/752cd404-5062-457f-9223-152507394688-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-2c4kj\" (UID: \"752cd404-5062-457f-9223-152507394688\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj"
Jul 17 12:58:54.411128 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.411101    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/752cd404-5062-457f-9223-152507394688-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-2c4kj\" (UID: \"752cd404-5062-457f-9223-152507394688\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj"
Jul 17 12:58:54.412781 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.412757    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/752cd404-5062-457f-9223-152507394688-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-2c4kj\" (UID: \"752cd404-5062-457f-9223-152507394688\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj"
Jul 17 12:58:54.419932 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.419910    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrdbb\" (UniqueName: \"kubernetes.io/projected/752cd404-5062-457f-9223-152507394688-kube-api-access-nrdbb\") pod \"message-dumper-predictor-6575bcfdfd-2c4kj\" (UID: \"752cd404-5062-457f-9223-152507394688\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj"
Jul 17 12:58:54.499132 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.499092    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj"
Jul 17 12:58:54.632978 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.632950    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj"]
Jul 17 12:58:54.634979 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:58:54.634953    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752cd404_5062_457f_9223_152507394688.slice/crio-2e9fef6f8d69c8d04d8eb86c7f15e0353bd35ae21a8ff8add722da721a2ada32 WatchSource:0}: Error finding container 2e9fef6f8d69c8d04d8eb86c7f15e0353bd35ae21a8ff8add722da721a2ada32: Status 404 returned error can't find the container with id 2e9fef6f8d69c8d04d8eb86c7f15e0353bd35ae21a8ff8add722da721a2ada32
Jul 17 12:58:54.636992 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.636976    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 12:58:54.768445 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.768335    2564 generic.go:356] "Generic (PLEG): container finished" podID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerID="d06d847825146a58083fe6cefed1e9f7c03a9c9867b0822969dc6797a15160d3" exitCode=2
Jul 17 12:58:54.768445 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.768399    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" event={"ID":"e0e9ad9e-af60-4f29-88f1-96dda14d647e","Type":"ContainerDied","Data":"d06d847825146a58083fe6cefed1e9f7c03a9c9867b0822969dc6797a15160d3"}
Jul 17 12:58:54.769647 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:54.769614    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj" event={"ID":"752cd404-5062-457f-9223-152507394688","Type":"ContainerStarted","Data":"2e9fef6f8d69c8d04d8eb86c7f15e0353bd35ae21a8ff8add722da721a2ada32"}
Jul 17 12:58:55.303404 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:55.303371    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:58:56.385662 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:56.385615    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 17 12:58:56.779993 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:56.779957    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj" event={"ID":"752cd404-5062-457f-9223-152507394688","Type":"ContainerStarted","Data":"8fab9a65011ce7c8faf06aa161105c68de1bc3eac14210da3df59f3e9ccf8d3e"}
Jul 17 12:58:56.780199 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:56.780025    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj" event={"ID":"752cd404-5062-457f-9223-152507394688","Type":"ContainerStarted","Data":"f9c13a57858951e827e21dee8ccc00b36565b79f285fe61b57c59e8c3d7abbba"}
Jul 17 12:58:56.780199 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:56.780175    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj"
Jul 17 12:58:56.801476 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:56.801431    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj" podStartSLOduration=1.6396833979999998 podStartE2EDuration="2.801418075s" totalImagesPullingTime="1.161734677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:58:56.799451512 +0000 UTC m=+1746.379563551" watchObservedRunningTime="2026-07-17 12:58:56.801418075 +0000 UTC m=+1746.381530092"
Jul 17 12:58:57.784334 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:57.784303    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj"
Jul 17 12:58:57.786212 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:57.786189    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj"
Jul 17 12:58:59.795184 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:59.795148    2564 generic.go:356] "Generic (PLEG): container finished" podID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerID="275f0527abd7bbac01a174f6de479d7778ea520e6c09fd6db7667279c6bd489a" exitCode=0
Jul 17 12:58:59.795648 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:58:59.795219    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" event={"ID":"e0e9ad9e-af60-4f29-88f1-96dda14d647e","Type":"ContainerDied","Data":"275f0527abd7bbac01a174f6de479d7778ea520e6c09fd6db7667279c6bd489a"}
Jul 17 12:59:01.385899 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:01.385851    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 17 12:59:01.390286 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:01.390247    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused"
Jul 17 12:59:01.390615 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:01.390585    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:59:04.799897 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:04.799867    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj"
Jul 17 12:59:06.385096 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:06.385048    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 17 12:59:06.385570 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:06.385198    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"
Jul 17 12:59:11.385129 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:11.385082    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 17 12:59:11.390519 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:11.390488    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused"
Jul 17 12:59:11.390890 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:11.390865    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:59:14.197751 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:14.197687    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:59:14.247158 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:14.247114    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"]
Jul 17 12:59:14.251082 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:14.251055    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"
Jul 17 12:59:14.254475 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:14.254448    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 17 12:59:14.254625 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:14.254507    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 17 12:59:14.267267 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:14.267225    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"]
Jul 17 12:59:14.283509 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:14.283473    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a1ff06b-005d-4538-bc3b-d73d6d729e44-kserve-provision-location\") pod \"isvc-logger-predictor-84dd4b499f-8sqxq\" (UID: \"4a1ff06b-005d-4538-bc3b-d73d6d729e44\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"
Jul 17 12:59:14.283683 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:14.283521    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a1ff06b-005d-4538-bc3b-d73d6d729e44-proxy-tls\") pod \"isvc-logger-predictor-84dd4b499f-8sqxq\" (UID: \"4a1ff06b-005d-4538-bc3b-d73d6d729e44\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"
Jul 17 12:59:14.283683 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:14.283562    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a1ff06b-005d-4538-bc3b-d73d6d729e44-cabundle-cert\") pod \"isvc-logger-predictor-84dd4b499f-8sqxq\" (UID: \"4a1ff06b-005d-4538-bc3b-d73d6d729e44\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"
Jul 17 12:59:14.283683 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:14.283577    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2dzr\" (UniqueName: \"kubernetes.io/projected/4a1ff06b-005d-4538-bc3b-d73d6d729e44-kube-api-access-q2dzr\") pod \"isvc-logger-predictor-84dd4b499f-8sqxq\" (UID: \"4a1ff06b-005d-4538-bc3b-d73d6d729e44\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"
Jul 17 12:59:14.283683 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:14.283603    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a1ff06b-005d-4538-bc3b-d73d6d729e44-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-84dd4b499f-8sqxq\" (UID: \"4a1ff06b-005d-4538-bc3b-d73d6d729e44\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"
Jul 17 12:59:14.384925 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:14.384880    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a1ff06b-005d-4538-bc3b-d73d6d729e44-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-84dd4b499f-8sqxq\" (UID: \"4a1ff06b-005d-4538-bc3b-d73d6d729e44\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"
Jul 17 12:59:14.385133 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:14.384954    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a1ff06b-005d-4538-bc3b-d73d6d729e44-kserve-provision-location\") pod \"isvc-logger-predictor-84dd4b499f-8sqxq\" (UID: \"4a1ff06b-005d-4538-bc3b-d73d6d729e44\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"
Jul 17 12:59:14.385133 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:14.384998    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a1ff06b-005d-4538-bc3b-d73d6d729e44-proxy-tls\") pod \"isvc-logger-predictor-84dd4b499f-8sqxq\" (UID: \"4a1ff06b-005d-4538-bc3b-d73d6d729e44\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"
Jul 17 12:59:14.385133 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:59:14.385123    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Jul 17 12:59:14.385293 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:14.385167    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a1ff06b-005d-4538-bc3b-d73d6d729e44-cabundle-cert\") pod \"isvc-logger-predictor-84dd4b499f-8sqxq\" (UID: \"4a1ff06b-005d-4538-bc3b-d73d6d729e44\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"
Jul 17 12:59:14.385293 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:59:14.385189    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a1ff06b-005d-4538-bc3b-d73d6d729e44-proxy-tls podName:4a1ff06b-005d-4538-bc3b-d73d6d729e44 nodeName:}" failed. No retries permitted until 2026-07-17 12:59:14.885166055 +0000 UTC m=+1764.465278055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4a1ff06b-005d-4538-bc3b-d73d6d729e44-proxy-tls") pod "isvc-logger-predictor-84dd4b499f-8sqxq" (UID: "4a1ff06b-005d-4538-bc3b-d73d6d729e44") : secret "isvc-logger-predictor-serving-cert" not found
Jul 17 12:59:14.385293 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:14.385219    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2dzr\" (UniqueName: \"kubernetes.io/projected/4a1ff06b-005d-4538-bc3b-d73d6d729e44-kube-api-access-q2dzr\") pod \"isvc-logger-predictor-84dd4b499f-8sqxq\" (UID: \"4a1ff06b-005d-4538-bc3b-d73d6d729e44\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"
Jul 17 12:59:14.385461 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:14.385367    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a1ff06b-005d-4538-bc3b-d73d6d729e44-kserve-provision-location\") pod \"isvc-logger-predictor-84dd4b499f-8sqxq\" (UID: \"4a1ff06b-005d-4538-bc3b-d73d6d729e44\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"
Jul 17 12:59:14.385672 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:14.385652    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a1ff06b-005d-4538-bc3b-d73d6d729e44-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-84dd4b499f-8sqxq\" (UID: \"4a1ff06b-005d-4538-bc3b-d73d6d729e44\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"
Jul 17 12:59:14.385835 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:14.385818    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a1ff06b-005d-4538-bc3b-d73d6d729e44-cabundle-cert\") pod \"isvc-logger-predictor-84dd4b499f-8sqxq\" (UID: \"4a1ff06b-005d-4538-bc3b-d73d6d729e44\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"
Jul 17 12:59:14.394846 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:14.394821    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2dzr\" (UniqueName: \"kubernetes.io/projected/4a1ff06b-005d-4538-bc3b-d73d6d729e44-kube-api-access-q2dzr\") pod \"isvc-logger-predictor-84dd4b499f-8sqxq\" (UID: \"4a1ff06b-005d-4538-bc3b-d73d6d729e44\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"
Jul 17 12:59:14.677192 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:14.677089    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:59:14.890093 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:14.890053    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a1ff06b-005d-4538-bc3b-d73d6d729e44-proxy-tls\") pod \"isvc-logger-predictor-84dd4b499f-8sqxq\" (UID: \"4a1ff06b-005d-4538-bc3b-d73d6d729e44\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"
Jul 17 12:59:14.892535 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:14.892506    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a1ff06b-005d-4538-bc3b-d73d6d729e44-proxy-tls\") pod \"isvc-logger-predictor-84dd4b499f-8sqxq\" (UID: \"4a1ff06b-005d-4538-bc3b-d73d6d729e44\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"
Jul 17 12:59:15.161666 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:15.161623    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"
Jul 17 12:59:15.302464 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:15.302439    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"]
Jul 17 12:59:15.304474 ip-10-0-137-99 kubenswrapper[2564]: W0717 12:59:15.304440    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a1ff06b_005d_4538_bc3b_d73d6d729e44.slice/crio-8887ed367aade93d3051637dc5a427481f50e29f6837528e0a814fb239a2b6ad WatchSource:0}: Error finding container 8887ed367aade93d3051637dc5a427481f50e29f6837528e0a814fb239a2b6ad: Status 404 returned error can't find the container with id 8887ed367aade93d3051637dc5a427481f50e29f6837528e0a814fb239a2b6ad
Jul 17 12:59:15.860679 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:15.860637    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" event={"ID":"4a1ff06b-005d-4538-bc3b-d73d6d729e44","Type":"ContainerStarted","Data":"7ef0250af5c582a92b6a0c481a19ce2ada16f89689508317cd4cc1b94bca66b3"}
Jul 17 12:59:15.860840 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:15.860751    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" event={"ID":"4a1ff06b-005d-4538-bc3b-d73d6d729e44","Type":"ContainerStarted","Data":"8887ed367aade93d3051637dc5a427481f50e29f6837528e0a814fb239a2b6ad"}
Jul 17 12:59:16.385934 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:16.385891    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 17 12:59:18.875070 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:18.875036    2564 generic.go:356] "Generic (PLEG): container finished" podID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerID="7ef0250af5c582a92b6a0c481a19ce2ada16f89689508317cd4cc1b94bca66b3" exitCode=0
Jul 17 12:59:18.875506 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:18.875078    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" event={"ID":"4a1ff06b-005d-4538-bc3b-d73d6d729e44","Type":"ContainerDied","Data":"7ef0250af5c582a92b6a0c481a19ce2ada16f89689508317cd4cc1b94bca66b3"}
Jul 17 12:59:19.882280 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:19.882248    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" event={"ID":"4a1ff06b-005d-4538-bc3b-d73d6d729e44","Type":"ContainerStarted","Data":"cf3f75732b29e053d1210ad15415fc00bc196d839bdc21103aefbc15e3b60392"}
Jul 17 12:59:19.882685 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:19.882320    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" event={"ID":"4a1ff06b-005d-4538-bc3b-d73d6d729e44","Type":"ContainerStarted","Data":"e05150f43e224632c2f958f973da1ba4496ad3e45189bc89d7d5752d5c883c04"}
Jul 17 12:59:19.882685 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:19.882349    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" event={"ID":"4a1ff06b-005d-4538-bc3b-d73d6d729e44","Type":"ContainerStarted","Data":"d4c6ea5f79d6c6052406bab7cfe9b465188bb3d94dda8a6d00b49be3276e06e4"}
Jul 17 12:59:19.882685 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:19.882649    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"
Jul 17 12:59:19.882834 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:19.882818    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"
Jul 17 12:59:19.884144 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:19.884116    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 17 12:59:19.904383 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:19.904329    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podStartSLOduration=2.422086246 podStartE2EDuration="5.904312467s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.482226221s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:59:19.902195923 +0000 UTC m=+1769.482307942" watchObservedRunningTime="2026-07-17 12:59:19.904312467 +0000 UTC m=+1769.484424485"
Jul 17 12:59:20.887040 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:20.887000    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 17 12:59:20.887040 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:20.887037    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"
Jul 17 12:59:20.888166 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:20.888141    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:59:21.386025 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:21.385985    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 17 12:59:21.390373 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:21.390342    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused"
Jul 17 12:59:21.390491 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:21.390465    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"
Jul 17 12:59:21.390685 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:21.390666    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:59:21.390786 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:21.390772    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"
Jul 17 12:59:21.891710 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:21.891654    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 17 12:59:21.892225 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:21.892202    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:59:24.256242 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.256216    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"
Jul 17 12:59:24.371046 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.370946    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0e9ad9e-af60-4f29-88f1-96dda14d647e-kserve-provision-location\") pod \"e0e9ad9e-af60-4f29-88f1-96dda14d647e\" (UID: \"e0e9ad9e-af60-4f29-88f1-96dda14d647e\") "
Jul 17 12:59:24.371220 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.371058    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0e9ad9e-af60-4f29-88f1-96dda14d647e-proxy-tls\") pod \"e0e9ad9e-af60-4f29-88f1-96dda14d647e\" (UID: \"e0e9ad9e-af60-4f29-88f1-96dda14d647e\") "
Jul 17 12:59:24.371220 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.371089    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0e9ad9e-af60-4f29-88f1-96dda14d647e-cabundle-cert\") pod \"e0e9ad9e-af60-4f29-88f1-96dda14d647e\" (UID: \"e0e9ad9e-af60-4f29-88f1-96dda14d647e\") "
Jul 17 12:59:24.371220 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.371107    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0e9ad9e-af60-4f29-88f1-96dda14d647e-kube-rbac-proxy-sar-config\") pod \"e0e9ad9e-af60-4f29-88f1-96dda14d647e\" (UID: \"e0e9ad9e-af60-4f29-88f1-96dda14d647e\") "
Jul 17 12:59:24.371220 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.371133    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5pdb\" (UniqueName: \"kubernetes.io/projected/e0e9ad9e-af60-4f29-88f1-96dda14d647e-kube-api-access-l5pdb\") pod \"e0e9ad9e-af60-4f29-88f1-96dda14d647e\" (UID: \"e0e9ad9e-af60-4f29-88f1-96dda14d647e\") "
Jul 17 12:59:24.371431 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.371321    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0e9ad9e-af60-4f29-88f1-96dda14d647e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e0e9ad9e-af60-4f29-88f1-96dda14d647e" (UID: "e0e9ad9e-af60-4f29-88f1-96dda14d647e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:59:24.371557 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.371522    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0e9ad9e-af60-4f29-88f1-96dda14d647e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e0e9ad9e-af60-4f29-88f1-96dda14d647e" (UID: "e0e9ad9e-af60-4f29-88f1-96dda14d647e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:59:24.371557 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.371526    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0e9ad9e-af60-4f29-88f1-96dda14d647e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e0e9ad9e-af60-4f29-88f1-96dda14d647e" (UID: "e0e9ad9e-af60-4f29-88f1-96dda14d647e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:59:24.373192 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.373159    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0e9ad9e-af60-4f29-88f1-96dda14d647e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e0e9ad9e-af60-4f29-88f1-96dda14d647e" (UID: "e0e9ad9e-af60-4f29-88f1-96dda14d647e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:59:24.373298 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.373273    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e9ad9e-af60-4f29-88f1-96dda14d647e-kube-api-access-l5pdb" (OuterVolumeSpecName: "kube-api-access-l5pdb") pod "e0e9ad9e-af60-4f29-88f1-96dda14d647e" (UID: "e0e9ad9e-af60-4f29-88f1-96dda14d647e"). InnerVolumeSpecName "kube-api-access-l5pdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:59:24.471935 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.471898    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0e9ad9e-af60-4f29-88f1-96dda14d647e-cabundle-cert\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:59:24.471935 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.471928    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0e9ad9e-af60-4f29-88f1-96dda14d647e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:59:24.471935 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.471938    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5pdb\" (UniqueName: \"kubernetes.io/projected/e0e9ad9e-af60-4f29-88f1-96dda14d647e-kube-api-access-l5pdb\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:59:24.471935 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.471948    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0e9ad9e-af60-4f29-88f1-96dda14d647e-kserve-provision-location\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:59:24.472183 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.471957    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0e9ad9e-af60-4f29-88f1-96dda14d647e-proxy-tls\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 12:59:24.906208 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.906168    2564 generic.go:356] "Generic (PLEG): container finished" podID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerID="dbeecc078d40133bde3b11e10080d945febbf9239749a542307fe6b8ab1acbd1" exitCode=0
Jul 17 12:59:24.906409 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.906240    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"
Jul 17 12:59:24.906409 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.906250    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" event={"ID":"e0e9ad9e-af60-4f29-88f1-96dda14d647e","Type":"ContainerDied","Data":"dbeecc078d40133bde3b11e10080d945febbf9239749a542307fe6b8ab1acbd1"}
Jul 17 12:59:24.906409 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.906342    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn" event={"ID":"e0e9ad9e-af60-4f29-88f1-96dda14d647e","Type":"ContainerDied","Data":"99acd60ad2c43aa1c7c1507848dcd1342a9b91cc55b1bb32dea1bb19125c6f81"}
Jul 17 12:59:24.906409 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.906376    2564 scope.go:117] "RemoveContainer" containerID="dbeecc078d40133bde3b11e10080d945febbf9239749a542307fe6b8ab1acbd1"
Jul 17 12:59:24.915667 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.915647    2564 scope.go:117] "RemoveContainer" containerID="d06d847825146a58083fe6cefed1e9f7c03a9c9867b0822969dc6797a15160d3"
Jul 17 12:59:24.923461 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.923441    2564 scope.go:117] "RemoveContainer" containerID="275f0527abd7bbac01a174f6de479d7778ea520e6c09fd6db7667279c6bd489a"
Jul 17 12:59:24.930717 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.930676    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"]
Jul 17 12:59:24.932989 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.932958    2564 scope.go:117] "RemoveContainer" containerID="d8a3bbce2a3801aba179d066b73c59d3b1c9148e15d35e585af8243cbba1367e"
Jul 17 12:59:24.935010 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.934990    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58f64b96c-6mnfn"]
Jul 17 12:59:24.940833 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.940812    2564 scope.go:117] "RemoveContainer" containerID="dbeecc078d40133bde3b11e10080d945febbf9239749a542307fe6b8ab1acbd1"
Jul 17 12:59:24.941144 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:59:24.941122    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbeecc078d40133bde3b11e10080d945febbf9239749a542307fe6b8ab1acbd1\": container with ID starting with dbeecc078d40133bde3b11e10080d945febbf9239749a542307fe6b8ab1acbd1 not found: ID does not exist" containerID="dbeecc078d40133bde3b11e10080d945febbf9239749a542307fe6b8ab1acbd1"
Jul 17 12:59:24.941197 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.941155    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbeecc078d40133bde3b11e10080d945febbf9239749a542307fe6b8ab1acbd1"} err="failed to get container status \"dbeecc078d40133bde3b11e10080d945febbf9239749a542307fe6b8ab1acbd1\": rpc error: code = NotFound desc = could not find container \"dbeecc078d40133bde3b11e10080d945febbf9239749a542307fe6b8ab1acbd1\": container with ID starting with dbeecc078d40133bde3b11e10080d945febbf9239749a542307fe6b8ab1acbd1 not found: ID does not exist"
Jul 17 12:59:24.941197 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.941177    2564 scope.go:117] "RemoveContainer" containerID="d06d847825146a58083fe6cefed1e9f7c03a9c9867b0822969dc6797a15160d3"
Jul 17 12:59:24.941459 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:59:24.941438    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d06d847825146a58083fe6cefed1e9f7c03a9c9867b0822969dc6797a15160d3\": container with ID starting with d06d847825146a58083fe6cefed1e9f7c03a9c9867b0822969dc6797a15160d3 not found: ID does not exist" containerID="d06d847825146a58083fe6cefed1e9f7c03a9c9867b0822969dc6797a15160d3"
Jul 17 12:59:24.941519 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.941465    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d06d847825146a58083fe6cefed1e9f7c03a9c9867b0822969dc6797a15160d3"} err="failed to get container status \"d06d847825146a58083fe6cefed1e9f7c03a9c9867b0822969dc6797a15160d3\": rpc error: code = NotFound desc = could not find container \"d06d847825146a58083fe6cefed1e9f7c03a9c9867b0822969dc6797a15160d3\": container with ID starting with d06d847825146a58083fe6cefed1e9f7c03a9c9867b0822969dc6797a15160d3 not found: ID does not exist"
Jul 17 12:59:24.941519 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.941485    2564 scope.go:117] "RemoveContainer" containerID="275f0527abd7bbac01a174f6de479d7778ea520e6c09fd6db7667279c6bd489a"
Jul 17 12:59:24.941737 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:59:24.941720    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"275f0527abd7bbac01a174f6de479d7778ea520e6c09fd6db7667279c6bd489a\": container with ID starting with 275f0527abd7bbac01a174f6de479d7778ea520e6c09fd6db7667279c6bd489a not found: ID does not exist" containerID="275f0527abd7bbac01a174f6de479d7778ea520e6c09fd6db7667279c6bd489a"
Jul 17 12:59:24.941803 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.941741    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"275f0527abd7bbac01a174f6de479d7778ea520e6c09fd6db7667279c6bd489a"} err="failed to get container status \"275f0527abd7bbac01a174f6de479d7778ea520e6c09fd6db7667279c6bd489a\": rpc error: code = NotFound desc = could not find container \"275f0527abd7bbac01a174f6de479d7778ea520e6c09fd6db7667279c6bd489a\": container with ID starting with 275f0527abd7bbac01a174f6de479d7778ea520e6c09fd6db7667279c6bd489a not found: ID does not exist"
Jul 17 12:59:24.941803 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.941758    2564 scope.go:117] "RemoveContainer" containerID="d8a3bbce2a3801aba179d066b73c59d3b1c9148e15d35e585af8243cbba1367e"
Jul 17 12:59:24.941988 ip-10-0-137-99 kubenswrapper[2564]: E0717 12:59:24.941971    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8a3bbce2a3801aba179d066b73c59d3b1c9148e15d35e585af8243cbba1367e\": container with ID starting with d8a3bbce2a3801aba179d066b73c59d3b1c9148e15d35e585af8243cbba1367e not found: ID does not exist" containerID="d8a3bbce2a3801aba179d066b73c59d3b1c9148e15d35e585af8243cbba1367e"
Jul 17 12:59:24.942037 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:24.941992    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a3bbce2a3801aba179d066b73c59d3b1c9148e15d35e585af8243cbba1367e"} err="failed to get container status \"d8a3bbce2a3801aba179d066b73c59d3b1c9148e15d35e585af8243cbba1367e\": rpc error: code = NotFound desc = could not find container \"d8a3bbce2a3801aba179d066b73c59d3b1c9148e15d35e585af8243cbba1367e\": container with ID starting with d8a3bbce2a3801aba179d066b73c59d3b1c9148e15d35e585af8243cbba1367e not found: ID does not exist"
Jul 17 12:59:25.014901 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:25.014858    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" path="/var/lib/kubelet/pods/e0e9ad9e-af60-4f29-88f1-96dda14d647e/volumes"
Jul 17 12:59:26.895766 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:26.895730    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"
Jul 17 12:59:26.896416 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:26.896368    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 17 12:59:26.896884 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:26.896847    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:59:36.896287 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:36.896245    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 17 12:59:36.896783 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:36.896651    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:59:39.890483 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:39.890449    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 12:59:46.896395 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:46.896298    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 17 12:59:46.896848 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:46.896732    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:59:50.906368 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:50.906338    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f958_7ab622df-86d3-4cc3-aba7-123f4bc0c996/ovn-acl-logging/0.log"
Jul 17 12:59:50.911738 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:50.911715    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f958_7ab622df-86d3-4cc3-aba7-123f4bc0c996/ovn-acl-logging/0.log"
Jul 17 12:59:56.896613 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:56.896554    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 17 12:59:56.897080 ip-10-0-137-99 kubenswrapper[2564]: I0717 12:59:56.897056    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 13:00:06.896645 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:06.896604    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 17 13:00:06.897192 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:06.897166    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 13:00:16.896694 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:16.896643    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 17 13:00:16.897212 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:16.897138    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 13:00:26.896776 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:26.896725    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 17 13:00:26.897268 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:26.897190    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 13:00:36.009874 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:36.009837    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"
Jul 17 13:00:36.010429 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:36.010127    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"
Jul 17 13:00:49.232313 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.232274    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-2c4kj_752cd404-5062-457f-9223-152507394688/kserve-container/0.log"
Jul 17 13:00:49.462093 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.462063    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"]
Jul 17 13:00:49.462417 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.462364    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="kserve-container" containerID="cri-o://d4c6ea5f79d6c6052406bab7cfe9b465188bb3d94dda8a6d00b49be3276e06e4" gracePeriod=30
Jul 17 13:00:49.462571 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.462393    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="agent" containerID="cri-o://cf3f75732b29e053d1210ad15415fc00bc196d839bdc21103aefbc15e3b60392" gracePeriod=30
Jul 17 13:00:49.462571 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.462409    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="kube-rbac-proxy" containerID="cri-o://e05150f43e224632c2f958f973da1ba4496ad3e45189bc89d7d5752d5c883c04" gracePeriod=30
Jul 17 13:00:49.474815 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.474770    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:00:49.536516 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.536483    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598"]
Jul 17 13:00:49.537014 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.536994    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="agent"
Jul 17 13:00:49.537014 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.537016    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="agent"
Jul 17 13:00:49.537192 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.537041    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="storage-initializer"
Jul 17 13:00:49.537192 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.537050    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="storage-initializer"
Jul 17 13:00:49.537192 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.537073    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="kserve-container"
Jul 17 13:00:49.537192 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.537082    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="kserve-container"
Jul 17 13:00:49.537192 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.537095    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="kube-rbac-proxy"
Jul 17 13:00:49.537192 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.537104    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="kube-rbac-proxy"
Jul 17 13:00:49.537367 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.537207    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="kserve-container"
Jul 17 13:00:49.537367 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.537220    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="kube-rbac-proxy"
Jul 17 13:00:49.537367 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.537230    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0e9ad9e-af60-4f29-88f1-96dda14d647e" containerName="agent"
Jul 17 13:00:49.541117 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.541094    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598"
Jul 17 13:00:49.543808 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.543786    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 17 13:00:49.544952 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.544921    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 17 13:00:49.552640 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.552547    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598"]
Jul 17 13:00:49.554847 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.554824    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj"]
Jul 17 13:00:49.555063 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.555040    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj" podUID="752cd404-5062-457f-9223-152507394688" containerName="kserve-container" containerID="cri-o://f9c13a57858951e827e21dee8ccc00b36565b79f285fe61b57c59e8c3d7abbba" gracePeriod=30
Jul 17 13:00:49.555158 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.555127    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj" podUID="752cd404-5062-457f-9223-152507394688" containerName="kube-rbac-proxy" containerID="cri-o://8fab9a65011ce7c8faf06aa161105c68de1bc3eac14210da3df59f3e9ccf8d3e" gracePeriod=30
Jul 17 13:00:49.613618 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.613582    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lpnd\" (UniqueName: \"kubernetes.io/projected/4f1d11c3-b0a7-4f83-b161-232673bf74f9-kube-api-access-6lpnd\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-hb598\" (UID: \"4f1d11c3-b0a7-4f83-b161-232673bf74f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598"
Jul 17 13:00:49.613821 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.613644    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f1d11c3-b0a7-4f83-b161-232673bf74f9-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-hb598\" (UID: \"4f1d11c3-b0a7-4f83-b161-232673bf74f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598"
Jul 17 13:00:49.613821 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.613735    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f1d11c3-b0a7-4f83-b161-232673bf74f9-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-hb598\" (UID: \"4f1d11c3-b0a7-4f83-b161-232673bf74f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598"
Jul 17 13:00:49.613821 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.613788    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f1d11c3-b0a7-4f83-b161-232673bf74f9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-hb598\" (UID: \"4f1d11c3-b0a7-4f83-b161-232673bf74f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598"
Jul 17 13:00:49.613821 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.613810    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f1d11c3-b0a7-4f83-b161-232673bf74f9-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-hb598\" (UID: \"4f1d11c3-b0a7-4f83-b161-232673bf74f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598"
Jul 17 13:00:49.714262 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.714224    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f1d11c3-b0a7-4f83-b161-232673bf74f9-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-hb598\" (UID: \"4f1d11c3-b0a7-4f83-b161-232673bf74f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598"
Jul 17 13:00:49.714426 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.714309    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f1d11c3-b0a7-4f83-b161-232673bf74f9-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-hb598\" (UID: \"4f1d11c3-b0a7-4f83-b161-232673bf74f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598"
Jul 17 13:00:49.714426 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.714346    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f1d11c3-b0a7-4f83-b161-232673bf74f9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-hb598\" (UID: \"4f1d11c3-b0a7-4f83-b161-232673bf74f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598"
Jul 17 13:00:49.714426 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.714375    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f1d11c3-b0a7-4f83-b161-232673bf74f9-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-hb598\" (UID: \"4f1d11c3-b0a7-4f83-b161-232673bf74f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598"
Jul 17 13:00:49.714426 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.714409    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lpnd\" (UniqueName: \"kubernetes.io/projected/4f1d11c3-b0a7-4f83-b161-232673bf74f9-kube-api-access-6lpnd\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-hb598\" (UID: \"4f1d11c3-b0a7-4f83-b161-232673bf74f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598"
Jul 17 13:00:49.714951 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.714919    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f1d11c3-b0a7-4f83-b161-232673bf74f9-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-hb598\" (UID: \"4f1d11c3-b0a7-4f83-b161-232673bf74f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598"
Jul 17 13:00:49.715109 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.714998    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f1d11c3-b0a7-4f83-b161-232673bf74f9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-hb598\" (UID: \"4f1d11c3-b0a7-4f83-b161-232673bf74f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598"
Jul 17 13:00:49.715109 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.715036    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f1d11c3-b0a7-4f83-b161-232673bf74f9-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-hb598\" (UID: \"4f1d11c3-b0a7-4f83-b161-232673bf74f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598"
Jul 17 13:00:49.716993 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.716972    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f1d11c3-b0a7-4f83-b161-232673bf74f9-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-hb598\" (UID: \"4f1d11c3-b0a7-4f83-b161-232673bf74f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598"
Jul 17 13:00:49.723941 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.723915    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lpnd\" (UniqueName: \"kubernetes.io/projected/4f1d11c3-b0a7-4f83-b161-232673bf74f9-kube-api-access-6lpnd\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-hb598\" (UID: \"4f1d11c3-b0a7-4f83-b161-232673bf74f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598"
Jul 17 13:00:49.797383 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.797352    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj"
Jul 17 13:00:49.853669 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.853625    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598"
Jul 17 13:00:49.915754 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.915588    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/752cd404-5062-457f-9223-152507394688-proxy-tls\") pod \"752cd404-5062-457f-9223-152507394688\" (UID: \"752cd404-5062-457f-9223-152507394688\") "
Jul 17 13:00:49.915754 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.915619    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/752cd404-5062-457f-9223-152507394688-kube-rbac-proxy-sar-config\") pod \"752cd404-5062-457f-9223-152507394688\" (UID: \"752cd404-5062-457f-9223-152507394688\") "
Jul 17 13:00:49.915754 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.915646    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrdbb\" (UniqueName: \"kubernetes.io/projected/752cd404-5062-457f-9223-152507394688-kube-api-access-nrdbb\") pod \"752cd404-5062-457f-9223-152507394688\" (UID: \"752cd404-5062-457f-9223-152507394688\") "
Jul 17 13:00:49.916167 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.916144    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/752cd404-5062-457f-9223-152507394688-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "752cd404-5062-457f-9223-152507394688" (UID: "752cd404-5062-457f-9223-152507394688"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:00:49.917568 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.917521    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/752cd404-5062-457f-9223-152507394688-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "752cd404-5062-457f-9223-152507394688" (UID: "752cd404-5062-457f-9223-152507394688"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:00:49.917845 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.917820    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752cd404-5062-457f-9223-152507394688-kube-api-access-nrdbb" (OuterVolumeSpecName: "kube-api-access-nrdbb") pod "752cd404-5062-457f-9223-152507394688" (UID: "752cd404-5062-457f-9223-152507394688"). InnerVolumeSpecName "kube-api-access-nrdbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:00:49.990764 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:49.990731    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598"]
Jul 17 13:00:49.992693 ip-10-0-137-99 kubenswrapper[2564]: W0717 13:00:49.992668    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f1d11c3_b0a7_4f83_b161_232673bf74f9.slice/crio-a5fde25ed7e69951f3960b615c6b34823fa442bf83075d84dace44317ba7440a WatchSource:0}: Error finding container a5fde25ed7e69951f3960b615c6b34823fa442bf83075d84dace44317ba7440a: Status 404 returned error can't find the container with id a5fde25ed7e69951f3960b615c6b34823fa442bf83075d84dace44317ba7440a
Jul 17 13:00:50.016414 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:50.016388    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/752cd404-5062-457f-9223-152507394688-proxy-tls\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:00:50.016414 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:50.016413    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/752cd404-5062-457f-9223-152507394688-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:00:50.016550 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:50.016425    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrdbb\" (UniqueName: \"kubernetes.io/projected/752cd404-5062-457f-9223-152507394688-kube-api-access-nrdbb\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:00:50.275052 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:50.275008    2564 generic.go:356] "Generic (PLEG): container finished" podID="752cd404-5062-457f-9223-152507394688" containerID="8fab9a65011ce7c8faf06aa161105c68de1bc3eac14210da3df59f3e9ccf8d3e" exitCode=2
Jul 17 13:00:50.275052 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:50.275045    2564 generic.go:356] "Generic (PLEG): container finished" podID="752cd404-5062-457f-9223-152507394688" containerID="f9c13a57858951e827e21dee8ccc00b36565b79f285fe61b57c59e8c3d7abbba" exitCode=2
Jul 17 13:00:50.275586 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:50.275074    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj"
Jul 17 13:00:50.275586 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:50.275081    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj" event={"ID":"752cd404-5062-457f-9223-152507394688","Type":"ContainerDied","Data":"8fab9a65011ce7c8faf06aa161105c68de1bc3eac14210da3df59f3e9ccf8d3e"}
Jul 17 13:00:50.275586 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:50.275162    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj" event={"ID":"752cd404-5062-457f-9223-152507394688","Type":"ContainerDied","Data":"f9c13a57858951e827e21dee8ccc00b36565b79f285fe61b57c59e8c3d7abbba"}
Jul 17 13:00:50.275586 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:50.275198    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj" event={"ID":"752cd404-5062-457f-9223-152507394688","Type":"ContainerDied","Data":"2e9fef6f8d69c8d04d8eb86c7f15e0353bd35ae21a8ff8add722da721a2ada32"}
Jul 17 13:00:50.275586 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:50.275235    2564 scope.go:117] "RemoveContainer" containerID="8fab9a65011ce7c8faf06aa161105c68de1bc3eac14210da3df59f3e9ccf8d3e"
Jul 17 13:00:50.276974 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:50.276952    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598" event={"ID":"4f1d11c3-b0a7-4f83-b161-232673bf74f9","Type":"ContainerStarted","Data":"43c48ff7f360a354da174973b51bc5a9ef173394734e93a0e4ce516d6225bf17"}
Jul 17 13:00:50.277130 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:50.277013    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598" event={"ID":"4f1d11c3-b0a7-4f83-b161-232673bf74f9","Type":"ContainerStarted","Data":"a5fde25ed7e69951f3960b615c6b34823fa442bf83075d84dace44317ba7440a"}
Jul 17 13:00:50.279972 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:50.279945    2564 generic.go:356] "Generic (PLEG): container finished" podID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerID="e05150f43e224632c2f958f973da1ba4496ad3e45189bc89d7d5752d5c883c04" exitCode=2
Jul 17 13:00:50.280081 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:50.280011    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" event={"ID":"4a1ff06b-005d-4538-bc3b-d73d6d729e44","Type":"ContainerDied","Data":"e05150f43e224632c2f958f973da1ba4496ad3e45189bc89d7d5752d5c883c04"}
Jul 17 13:00:50.284946 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:50.284925    2564 scope.go:117] "RemoveContainer" containerID="f9c13a57858951e827e21dee8ccc00b36565b79f285fe61b57c59e8c3d7abbba"
Jul 17 13:00:50.292257 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:50.292243    2564 scope.go:117] "RemoveContainer" containerID="8fab9a65011ce7c8faf06aa161105c68de1bc3eac14210da3df59f3e9ccf8d3e"
Jul 17 13:00:50.292484 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:00:50.292465    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fab9a65011ce7c8faf06aa161105c68de1bc3eac14210da3df59f3e9ccf8d3e\": container with ID starting with 8fab9a65011ce7c8faf06aa161105c68de1bc3eac14210da3df59f3e9ccf8d3e not found: ID does not exist" containerID="8fab9a65011ce7c8faf06aa161105c68de1bc3eac14210da3df59f3e9ccf8d3e"
Jul 17 13:00:50.292563 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:50.292494    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fab9a65011ce7c8faf06aa161105c68de1bc3eac14210da3df59f3e9ccf8d3e"} err="failed to get container status \"8fab9a65011ce7c8faf06aa161105c68de1bc3eac14210da3df59f3e9ccf8d3e\": rpc error: code = NotFound desc = could not find container \"8fab9a65011ce7c8faf06aa161105c68de1bc3eac14210da3df59f3e9ccf8d3e\": container with ID starting with 8fab9a65011ce7c8faf06aa161105c68de1bc3eac14210da3df59f3e9ccf8d3e not found: ID does not exist"
Jul 17 13:00:50.292563 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:50.292519    2564 scope.go:117] "RemoveContainer" containerID="f9c13a57858951e827e21dee8ccc00b36565b79f285fe61b57c59e8c3d7abbba"
Jul 17 13:00:50.292860 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:00:50.292824    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9c13a57858951e827e21dee8ccc00b36565b79f285fe61b57c59e8c3d7abbba\": container with ID starting with f9c13a57858951e827e21dee8ccc00b36565b79f285fe61b57c59e8c3d7abbba not found: ID does not exist" containerID="f9c13a57858951e827e21dee8ccc00b36565b79f285fe61b57c59e8c3d7abbba"
Jul 17 13:00:50.292860 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:50.292847    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9c13a57858951e827e21dee8ccc00b36565b79f285fe61b57c59e8c3d7abbba"} err="failed to get container status \"f9c13a57858951e827e21dee8ccc00b36565b79f285fe61b57c59e8c3d7abbba\": rpc error: code = NotFound desc = could not find container \"f9c13a57858951e827e21dee8ccc00b36565b79f285fe61b57c59e8c3d7abbba\": container with ID starting with f9c13a57858951e827e21dee8ccc00b36565b79f285fe61b57c59e8c3d7abbba not found: ID does not exist"
Jul 17 13:00:50.292960 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:50.292862    2564 scope.go:117] "RemoveContainer" containerID="8fab9a65011ce7c8faf06aa161105c68de1bc3eac14210da3df59f3e9ccf8d3e"
Jul 17 13:00:50.293122 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:50.293099    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fab9a65011ce7c8faf06aa161105c68de1bc3eac14210da3df59f3e9ccf8d3e"} err="failed to get container status \"8fab9a65011ce7c8faf06aa161105c68de1bc3eac14210da3df59f3e9ccf8d3e\": rpc error: code = NotFound desc = could not find container \"8fab9a65011ce7c8faf06aa161105c68de1bc3eac14210da3df59f3e9ccf8d3e\": container with ID starting with 8fab9a65011ce7c8faf06aa161105c68de1bc3eac14210da3df59f3e9ccf8d3e not found: ID does not exist"
Jul 17 13:00:50.293172 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:50.293125    2564 scope.go:117] "RemoveContainer" containerID="f9c13a57858951e827e21dee8ccc00b36565b79f285fe61b57c59e8c3d7abbba"
Jul 17 13:00:50.293352 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:50.293333    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9c13a57858951e827e21dee8ccc00b36565b79f285fe61b57c59e8c3d7abbba"} err="failed to get container status \"f9c13a57858951e827e21dee8ccc00b36565b79f285fe61b57c59e8c3d7abbba\": rpc error: code = NotFound desc = could not find container \"f9c13a57858951e827e21dee8ccc00b36565b79f285fe61b57c59e8c3d7abbba\": container with ID starting with f9c13a57858951e827e21dee8ccc00b36565b79f285fe61b57c59e8c3d7abbba not found: ID does not exist"
Jul 17 13:00:50.312220 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:50.312189    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj"]
Jul 17 13:00:50.313860 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:50.313835    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2c4kj"]
Jul 17 13:00:50.840421 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:50.840388    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:00:50.941599 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:50.941549    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:00:51.013442 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:51.013408    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="752cd404-5062-457f-9223-152507394688" path="/var/lib/kubelet/pods/752cd404-5062-457f-9223-152507394688/volumes"
Jul 17 13:00:51.892401 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:51.892346    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 17 13:00:54.300641 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:54.300609    2564 generic.go:356] "Generic (PLEG): container finished" podID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerID="d4c6ea5f79d6c6052406bab7cfe9b465188bb3d94dda8a6d00b49be3276e06e4" exitCode=0
Jul 17 13:00:54.301040 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:54.300688    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" event={"ID":"4a1ff06b-005d-4538-bc3b-d73d6d729e44","Type":"ContainerDied","Data":"d4c6ea5f79d6c6052406bab7cfe9b465188bb3d94dda8a6d00b49be3276e06e4"}
Jul 17 13:00:56.010094 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:56.010044    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 17 13:00:56.010591 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:56.010363    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 13:00:56.892673 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:00:56.892630    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 17 13:01:01.892716 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:01.892664    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 17 13:01:01.893183 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:01.892834    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"
Jul 17 13:01:06.009464 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:06.009420    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 17 13:01:06.009975 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:06.009861    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 13:01:06.892564 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:06.892523    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 17 13:01:11.892478 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:11.892431    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 17 13:01:16.010300 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:16.010199    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 17 13:01:16.010736 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:16.010350    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"
Jul 17 13:01:16.010736 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:16.010522    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 13:01:16.010736 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:16.010643    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"
Jul 17 13:01:16.892770 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:16.892727    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 17 13:01:19.404028 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:19.403995    2564 generic.go:356] "Generic (PLEG): container finished" podID="4f1d11c3-b0a7-4f83-b161-232673bf74f9" containerID="43c48ff7f360a354da174973b51bc5a9ef173394734e93a0e4ce516d6225bf17" exitCode=0
Jul 17 13:01:19.404406 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:19.404069    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598" event={"ID":"4f1d11c3-b0a7-4f83-b161-232673bf74f9","Type":"ContainerDied","Data":"43c48ff7f360a354da174973b51bc5a9ef173394734e93a0e4ce516d6225bf17"}
Jul 17 13:01:19.621773 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:19.621744    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"
Jul 17 13:01:19.762970 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:19.762891    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a1ff06b-005d-4538-bc3b-d73d6d729e44-kserve-provision-location\") pod \"4a1ff06b-005d-4538-bc3b-d73d6d729e44\" (UID: \"4a1ff06b-005d-4538-bc3b-d73d6d729e44\") "
Jul 17 13:01:19.762970 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:19.762931    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a1ff06b-005d-4538-bc3b-d73d6d729e44-proxy-tls\") pod \"4a1ff06b-005d-4538-bc3b-d73d6d729e44\" (UID: \"4a1ff06b-005d-4538-bc3b-d73d6d729e44\") "
Jul 17 13:01:19.763192 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:19.762977    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a1ff06b-005d-4538-bc3b-d73d6d729e44-cabundle-cert\") pod \"4a1ff06b-005d-4538-bc3b-d73d6d729e44\" (UID: \"4a1ff06b-005d-4538-bc3b-d73d6d729e44\") "
Jul 17 13:01:19.763192 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:19.763030    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2dzr\" (UniqueName: \"kubernetes.io/projected/4a1ff06b-005d-4538-bc3b-d73d6d729e44-kube-api-access-q2dzr\") pod \"4a1ff06b-005d-4538-bc3b-d73d6d729e44\" (UID: \"4a1ff06b-005d-4538-bc3b-d73d6d729e44\") "
Jul 17 13:01:19.763192 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:19.763071    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a1ff06b-005d-4538-bc3b-d73d6d729e44-kube-rbac-proxy-sar-config\") pod \"4a1ff06b-005d-4538-bc3b-d73d6d729e44\" (UID: \"4a1ff06b-005d-4538-bc3b-d73d6d729e44\") "
Jul 17 13:01:19.763346 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:19.763191    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a1ff06b-005d-4538-bc3b-d73d6d729e44-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4a1ff06b-005d-4538-bc3b-d73d6d729e44" (UID: "4a1ff06b-005d-4538-bc3b-d73d6d729e44"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 13:01:19.763346 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:19.763309    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a1ff06b-005d-4538-bc3b-d73d6d729e44-kserve-provision-location\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:01:19.763456 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:19.763405    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a1ff06b-005d-4538-bc3b-d73d6d729e44-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4a1ff06b-005d-4538-bc3b-d73d6d729e44" (UID: "4a1ff06b-005d-4538-bc3b-d73d6d729e44"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:01:19.763456 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:19.763409    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a1ff06b-005d-4538-bc3b-d73d6d729e44-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4a1ff06b-005d-4538-bc3b-d73d6d729e44" (UID: "4a1ff06b-005d-4538-bc3b-d73d6d729e44"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:01:19.765102 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:19.765074    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a1ff06b-005d-4538-bc3b-d73d6d729e44-kube-api-access-q2dzr" (OuterVolumeSpecName: "kube-api-access-q2dzr") pod "4a1ff06b-005d-4538-bc3b-d73d6d729e44" (UID: "4a1ff06b-005d-4538-bc3b-d73d6d729e44"). InnerVolumeSpecName "kube-api-access-q2dzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:01:19.765210 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:19.765126    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a1ff06b-005d-4538-bc3b-d73d6d729e44-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4a1ff06b-005d-4538-bc3b-d73d6d729e44" (UID: "4a1ff06b-005d-4538-bc3b-d73d6d729e44"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:01:19.864767 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:19.864694    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a1ff06b-005d-4538-bc3b-d73d6d729e44-cabundle-cert\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:01:19.864767 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:19.864753    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2dzr\" (UniqueName: \"kubernetes.io/projected/4a1ff06b-005d-4538-bc3b-d73d6d729e44-kube-api-access-q2dzr\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:01:19.864767 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:19.864768    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a1ff06b-005d-4538-bc3b-d73d6d729e44-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:01:19.865008 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:19.864781    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a1ff06b-005d-4538-bc3b-d73d6d729e44-proxy-tls\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:01:20.411638 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:20.411541    2564 generic.go:356] "Generic (PLEG): container finished" podID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerID="cf3f75732b29e053d1210ad15415fc00bc196d839bdc21103aefbc15e3b60392" exitCode=0
Jul 17 13:01:20.411638 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:20.411609    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"
Jul 17 13:01:20.412156 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:20.411625    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" event={"ID":"4a1ff06b-005d-4538-bc3b-d73d6d729e44","Type":"ContainerDied","Data":"cf3f75732b29e053d1210ad15415fc00bc196d839bdc21103aefbc15e3b60392"}
Jul 17 13:01:20.412156 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:20.411738    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq" event={"ID":"4a1ff06b-005d-4538-bc3b-d73d6d729e44","Type":"ContainerDied","Data":"8887ed367aade93d3051637dc5a427481f50e29f6837528e0a814fb239a2b6ad"}
Jul 17 13:01:20.412156 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:20.411793    2564 scope.go:117] "RemoveContainer" containerID="cf3f75732b29e053d1210ad15415fc00bc196d839bdc21103aefbc15e3b60392"
Jul 17 13:01:20.436990 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:20.436944    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"]
Jul 17 13:01:20.440661 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:20.440629    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-84dd4b499f-8sqxq"]
Jul 17 13:01:21.015433 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:21.015376    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" path="/var/lib/kubelet/pods/4a1ff06b-005d-4538-bc3b-d73d6d729e44/volumes"
Jul 17 13:01:21.173331 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:21.173298    2564 scope.go:117] "RemoveContainer" containerID="e05150f43e224632c2f958f973da1ba4496ad3e45189bc89d7d5752d5c883c04"
Jul 17 13:01:21.576469 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:21.576271    2564 scope.go:117] "RemoveContainer" containerID="d4c6ea5f79d6c6052406bab7cfe9b465188bb3d94dda8a6d00b49be3276e06e4"
Jul 17 13:01:21.677722 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:21.677682    2564 scope.go:117] "RemoveContainer" containerID="7ef0250af5c582a92b6a0c481a19ce2ada16f89689508317cd4cc1b94bca66b3"
Jul 17 13:01:21.778991 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:21.778965    2564 scope.go:117] "RemoveContainer" containerID="cf3f75732b29e053d1210ad15415fc00bc196d839bdc21103aefbc15e3b60392"
Jul 17 13:01:21.779462 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:01:21.779433    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf3f75732b29e053d1210ad15415fc00bc196d839bdc21103aefbc15e3b60392\": container with ID starting with cf3f75732b29e053d1210ad15415fc00bc196d839bdc21103aefbc15e3b60392 not found: ID does not exist" containerID="cf3f75732b29e053d1210ad15415fc00bc196d839bdc21103aefbc15e3b60392"
Jul 17 13:01:21.779559 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:21.779475    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf3f75732b29e053d1210ad15415fc00bc196d839bdc21103aefbc15e3b60392"} err="failed to get container status \"cf3f75732b29e053d1210ad15415fc00bc196d839bdc21103aefbc15e3b60392\": rpc error: code = NotFound desc = could not find container \"cf3f75732b29e053d1210ad15415fc00bc196d839bdc21103aefbc15e3b60392\": container with ID starting with cf3f75732b29e053d1210ad15415fc00bc196d839bdc21103aefbc15e3b60392 not found: ID does not exist"
Jul 17 13:01:21.779559 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:21.779503    2564 scope.go:117] "RemoveContainer" containerID="e05150f43e224632c2f958f973da1ba4496ad3e45189bc89d7d5752d5c883c04"
Jul 17 13:01:21.780214 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:01:21.780192    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e05150f43e224632c2f958f973da1ba4496ad3e45189bc89d7d5752d5c883c04\": container with ID starting with e05150f43e224632c2f958f973da1ba4496ad3e45189bc89d7d5752d5c883c04 not found: ID does not exist" containerID="e05150f43e224632c2f958f973da1ba4496ad3e45189bc89d7d5752d5c883c04"
Jul 17 13:01:21.780319 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:21.780223    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e05150f43e224632c2f958f973da1ba4496ad3e45189bc89d7d5752d5c883c04"} err="failed to get container status \"e05150f43e224632c2f958f973da1ba4496ad3e45189bc89d7d5752d5c883c04\": rpc error: code = NotFound desc = could not find container \"e05150f43e224632c2f958f973da1ba4496ad3e45189bc89d7d5752d5c883c04\": container with ID starting with e05150f43e224632c2f958f973da1ba4496ad3e45189bc89d7d5752d5c883c04 not found: ID does not exist"
Jul 17 13:01:21.780319 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:21.780250    2564 scope.go:117] "RemoveContainer" containerID="d4c6ea5f79d6c6052406bab7cfe9b465188bb3d94dda8a6d00b49be3276e06e4"
Jul 17 13:01:21.780594 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:01:21.780566    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4c6ea5f79d6c6052406bab7cfe9b465188bb3d94dda8a6d00b49be3276e06e4\": container with ID starting with d4c6ea5f79d6c6052406bab7cfe9b465188bb3d94dda8a6d00b49be3276e06e4 not found: ID does not exist" containerID="d4c6ea5f79d6c6052406bab7cfe9b465188bb3d94dda8a6d00b49be3276e06e4"
Jul 17 13:01:21.780722 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:21.780602    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4c6ea5f79d6c6052406bab7cfe9b465188bb3d94dda8a6d00b49be3276e06e4"} err="failed to get container status \"d4c6ea5f79d6c6052406bab7cfe9b465188bb3d94dda8a6d00b49be3276e06e4\": rpc error: code = NotFound desc = could not find container \"d4c6ea5f79d6c6052406bab7cfe9b465188bb3d94dda8a6d00b49be3276e06e4\": container with ID starting with d4c6ea5f79d6c6052406bab7cfe9b465188bb3d94dda8a6d00b49be3276e06e4 not found: ID does not exist"
Jul 17 13:01:21.780722 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:21.780623    2564 scope.go:117] "RemoveContainer" containerID="7ef0250af5c582a92b6a0c481a19ce2ada16f89689508317cd4cc1b94bca66b3"
Jul 17 13:01:21.781035 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:01:21.781010    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ef0250af5c582a92b6a0c481a19ce2ada16f89689508317cd4cc1b94bca66b3\": container with ID starting with 7ef0250af5c582a92b6a0c481a19ce2ada16f89689508317cd4cc1b94bca66b3 not found: ID does not exist" containerID="7ef0250af5c582a92b6a0c481a19ce2ada16f89689508317cd4cc1b94bca66b3"
Jul 17 13:01:21.781141 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:21.781045    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ef0250af5c582a92b6a0c481a19ce2ada16f89689508317cd4cc1b94bca66b3"} err="failed to get container status \"7ef0250af5c582a92b6a0c481a19ce2ada16f89689508317cd4cc1b94bca66b3\": rpc error: code = NotFound desc = could not find container \"7ef0250af5c582a92b6a0c481a19ce2ada16f89689508317cd4cc1b94bca66b3\": container with ID starting with 7ef0250af5c582a92b6a0c481a19ce2ada16f89689508317cd4cc1b94bca66b3 not found: ID does not exist"
Jul 17 13:01:23.058018 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:23.057986    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:01:23.119062 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:01:23.119013    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:02:10.219579 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:02:10.219543    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:02:10.279425 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:02:10.279368    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:02:33.701326 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:02:33.701283    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:02:33.802268 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:02:33.802216    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:03:20.989651 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:20.989614    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598" event={"ID":"4f1d11c3-b0a7-4f83-b161-232673bf74f9","Type":"ContainerStarted","Data":"652acd03da3f392d5580c1ef484f3746597b3bcc04fbd4edbaa928d3ca3184c6"}
Jul 17 13:03:20.990189 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:20.989685    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598" event={"ID":"4f1d11c3-b0a7-4f83-b161-232673bf74f9","Type":"ContainerStarted","Data":"0630f7caea84cdc2f01f2e365aa9c276d75e631161f21c3aeedd04a7fe800657"}
Jul 17 13:03:20.990189 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:20.989743    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598"
Jul 17 13:03:21.019296 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:21.019245    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598" podStartSLOduration=1.881212395 podStartE2EDuration="2m32.019228745s" totalImagesPullingTime="2m0.972731143s" totalInitContainerRuntime="29.165285207s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:00:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:03:21.016999493 +0000 UTC m=+2010.597111524" watchObservedRunningTime="2026-07-17 13:03:21.019228745 +0000 UTC m=+2010.599340763"
Jul 17 13:03:21.993788 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:21.993750    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598"
Jul 17 13:03:21.995087 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:21.995059    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598" podUID="4f1d11c3-b0a7-4f83-b161-232673bf74f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 17 13:03:22.997934 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:22.997886    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598" podUID="4f1d11c3-b0a7-4f83-b161-232673bf74f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 17 13:03:28.003440 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:28.003410    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598"
Jul 17 13:03:28.004285 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:28.004010    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598" podUID="4f1d11c3-b0a7-4f83-b161-232673bf74f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 17 13:03:38.005367 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:38.005334    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598"
Jul 17 13:03:39.697625 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:39.697565    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598"]
Jul 17 13:03:39.698095 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:39.697854    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598" podUID="4f1d11c3-b0a7-4f83-b161-232673bf74f9" containerName="kserve-container" containerID="cri-o://0630f7caea84cdc2f01f2e365aa9c276d75e631161f21c3aeedd04a7fe800657" gracePeriod=30
Jul 17 13:03:39.698095 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:39.697902    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598" podUID="4f1d11c3-b0a7-4f83-b161-232673bf74f9" containerName="kube-rbac-proxy" containerID="cri-o://652acd03da3f392d5580c1ef484f3746597b3bcc04fbd4edbaa928d3ca3184c6" gracePeriod=30
Jul 17 13:03:39.720608 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:39.720583    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:03:40.071858 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:40.071826    2564 generic.go:356] "Generic (PLEG): container finished" podID="4f1d11c3-b0a7-4f83-b161-232673bf74f9" containerID="652acd03da3f392d5580c1ef484f3746597b3bcc04fbd4edbaa928d3ca3184c6" exitCode=2
Jul 17 13:03:40.072029 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:40.071891    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598" event={"ID":"4f1d11c3-b0a7-4f83-b161-232673bf74f9","Type":"ContainerDied","Data":"652acd03da3f392d5580c1ef484f3746597b3bcc04fbd4edbaa928d3ca3184c6"}
Jul 17 13:03:40.686810 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:40.686691    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598"
Jul 17 13:03:40.855408 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:40.855368    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f1d11c3-b0a7-4f83-b161-232673bf74f9-kserve-provision-location\") pod \"4f1d11c3-b0a7-4f83-b161-232673bf74f9\" (UID: \"4f1d11c3-b0a7-4f83-b161-232673bf74f9\") "
Jul 17 13:03:40.855934 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:40.855431    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f1d11c3-b0a7-4f83-b161-232673bf74f9-kube-rbac-proxy-sar-config\") pod \"4f1d11c3-b0a7-4f83-b161-232673bf74f9\" (UID: \"4f1d11c3-b0a7-4f83-b161-232673bf74f9\") "
Jul 17 13:03:40.855934 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:40.855465    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lpnd\" (UniqueName: \"kubernetes.io/projected/4f1d11c3-b0a7-4f83-b161-232673bf74f9-kube-api-access-6lpnd\") pod \"4f1d11c3-b0a7-4f83-b161-232673bf74f9\" (UID: \"4f1d11c3-b0a7-4f83-b161-232673bf74f9\") "
Jul 17 13:03:40.855934 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:40.855487    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f1d11c3-b0a7-4f83-b161-232673bf74f9-proxy-tls\") pod \"4f1d11c3-b0a7-4f83-b161-232673bf74f9\" (UID: \"4f1d11c3-b0a7-4f83-b161-232673bf74f9\") "
Jul 17 13:03:40.855934 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:40.855507    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f1d11c3-b0a7-4f83-b161-232673bf74f9-cabundle-cert\") pod \"4f1d11c3-b0a7-4f83-b161-232673bf74f9\" (UID: \"4f1d11c3-b0a7-4f83-b161-232673bf74f9\") "
Jul 17 13:03:40.856152 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:40.855955    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f1d11c3-b0a7-4f83-b161-232673bf74f9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4f1d11c3-b0a7-4f83-b161-232673bf74f9" (UID: "4f1d11c3-b0a7-4f83-b161-232673bf74f9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:03:40.856152 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:40.856000    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f1d11c3-b0a7-4f83-b161-232673bf74f9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4f1d11c3-b0a7-4f83-b161-232673bf74f9" (UID: "4f1d11c3-b0a7-4f83-b161-232673bf74f9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:03:40.857710 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:40.857670    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f1d11c3-b0a7-4f83-b161-232673bf74f9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4f1d11c3-b0a7-4f83-b161-232673bf74f9" (UID: "4f1d11c3-b0a7-4f83-b161-232673bf74f9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:03:40.857830 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:40.857783    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f1d11c3-b0a7-4f83-b161-232673bf74f9-kube-api-access-6lpnd" (OuterVolumeSpecName: "kube-api-access-6lpnd") pod "4f1d11c3-b0a7-4f83-b161-232673bf74f9" (UID: "4f1d11c3-b0a7-4f83-b161-232673bf74f9"). InnerVolumeSpecName "kube-api-access-6lpnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:03:40.859719 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:40.859679    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f1d11c3-b0a7-4f83-b161-232673bf74f9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4f1d11c3-b0a7-4f83-b161-232673bf74f9" (UID: "4f1d11c3-b0a7-4f83-b161-232673bf74f9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 13:03:40.956554 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:40.956460    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f1d11c3-b0a7-4f83-b161-232673bf74f9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:03:40.956554 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:40.956493    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lpnd\" (UniqueName: \"kubernetes.io/projected/4f1d11c3-b0a7-4f83-b161-232673bf74f9-kube-api-access-6lpnd\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:03:40.956554 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:40.956507    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f1d11c3-b0a7-4f83-b161-232673bf74f9-proxy-tls\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:03:40.956554 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:40.956523    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f1d11c3-b0a7-4f83-b161-232673bf74f9-cabundle-cert\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:03:40.956554 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:40.956534    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f1d11c3-b0a7-4f83-b161-232673bf74f9-kserve-provision-location\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:03:41.078276 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:41.078240    2564 generic.go:356] "Generic (PLEG): container finished" podID="4f1d11c3-b0a7-4f83-b161-232673bf74f9" containerID="0630f7caea84cdc2f01f2e365aa9c276d75e631161f21c3aeedd04a7fe800657" exitCode=0
Jul 17 13:03:41.078453 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:41.078306    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598"
Jul 17 13:03:41.078453 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:41.078330    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598" event={"ID":"4f1d11c3-b0a7-4f83-b161-232673bf74f9","Type":"ContainerDied","Data":"0630f7caea84cdc2f01f2e365aa9c276d75e631161f21c3aeedd04a7fe800657"}
Jul 17 13:03:41.078453 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:41.078392    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598" event={"ID":"4f1d11c3-b0a7-4f83-b161-232673bf74f9","Type":"ContainerDied","Data":"a5fde25ed7e69951f3960b615c6b34823fa442bf83075d84dace44317ba7440a"}
Jul 17 13:03:41.078453 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:41.078419    2564 scope.go:117] "RemoveContainer" containerID="652acd03da3f392d5580c1ef484f3746597b3bcc04fbd4edbaa928d3ca3184c6"
Jul 17 13:03:41.086678 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:41.086658    2564 scope.go:117] "RemoveContainer" containerID="0630f7caea84cdc2f01f2e365aa9c276d75e631161f21c3aeedd04a7fe800657"
Jul 17 13:03:41.094438 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:41.094419    2564 scope.go:117] "RemoveContainer" containerID="43c48ff7f360a354da174973b51bc5a9ef173394734e93a0e4ce516d6225bf17"
Jul 17 13:03:41.097828 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:41.097802    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598"]
Jul 17 13:03:41.101644 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:41.101614    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-hb598"]
Jul 17 13:03:41.102920 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:41.102905    2564 scope.go:117] "RemoveContainer" containerID="652acd03da3f392d5580c1ef484f3746597b3bcc04fbd4edbaa928d3ca3184c6"
Jul 17 13:03:41.103240 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:03:41.103221    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"652acd03da3f392d5580c1ef484f3746597b3bcc04fbd4edbaa928d3ca3184c6\": container with ID starting with 652acd03da3f392d5580c1ef484f3746597b3bcc04fbd4edbaa928d3ca3184c6 not found: ID does not exist" containerID="652acd03da3f392d5580c1ef484f3746597b3bcc04fbd4edbaa928d3ca3184c6"
Jul 17 13:03:41.103312 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:41.103254    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"652acd03da3f392d5580c1ef484f3746597b3bcc04fbd4edbaa928d3ca3184c6"} err="failed to get container status \"652acd03da3f392d5580c1ef484f3746597b3bcc04fbd4edbaa928d3ca3184c6\": rpc error: code = NotFound desc = could not find container \"652acd03da3f392d5580c1ef484f3746597b3bcc04fbd4edbaa928d3ca3184c6\": container with ID starting with 652acd03da3f392d5580c1ef484f3746597b3bcc04fbd4edbaa928d3ca3184c6 not found: ID does not exist"
Jul 17 13:03:41.103312 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:41.103283    2564 scope.go:117] "RemoveContainer" containerID="0630f7caea84cdc2f01f2e365aa9c276d75e631161f21c3aeedd04a7fe800657"
Jul 17 13:03:41.103588 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:03:41.103571    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0630f7caea84cdc2f01f2e365aa9c276d75e631161f21c3aeedd04a7fe800657\": container with ID starting with 0630f7caea84cdc2f01f2e365aa9c276d75e631161f21c3aeedd04a7fe800657 not found: ID does not exist" containerID="0630f7caea84cdc2f01f2e365aa9c276d75e631161f21c3aeedd04a7fe800657"
Jul 17 13:03:41.103627 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:41.103594    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0630f7caea84cdc2f01f2e365aa9c276d75e631161f21c3aeedd04a7fe800657"} err="failed to get container status \"0630f7caea84cdc2f01f2e365aa9c276d75e631161f21c3aeedd04a7fe800657\": rpc error: code = NotFound desc = could not find container \"0630f7caea84cdc2f01f2e365aa9c276d75e631161f21c3aeedd04a7fe800657\": container with ID starting with 0630f7caea84cdc2f01f2e365aa9c276d75e631161f21c3aeedd04a7fe800657 not found: ID does not exist"
Jul 17 13:03:41.103627 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:41.103609    2564 scope.go:117] "RemoveContainer" containerID="43c48ff7f360a354da174973b51bc5a9ef173394734e93a0e4ce516d6225bf17"
Jul 17 13:03:41.103868 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:03:41.103851    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43c48ff7f360a354da174973b51bc5a9ef173394734e93a0e4ce516d6225bf17\": container with ID starting with 43c48ff7f360a354da174973b51bc5a9ef173394734e93a0e4ce516d6225bf17 not found: ID does not exist" containerID="43c48ff7f360a354da174973b51bc5a9ef173394734e93a0e4ce516d6225bf17"
Jul 17 13:03:41.103930 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:41.103875    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43c48ff7f360a354da174973b51bc5a9ef173394734e93a0e4ce516d6225bf17"} err="failed to get container status \"43c48ff7f360a354da174973b51bc5a9ef173394734e93a0e4ce516d6225bf17\": rpc error: code = NotFound desc = could not find container \"43c48ff7f360a354da174973b51bc5a9ef173394734e93a0e4ce516d6225bf17\": container with ID starting with 43c48ff7f360a354da174973b51bc5a9ef173394734e93a0e4ce516d6225bf17 not found: ID does not exist"
Jul 17 13:03:43.014059 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:43.014024    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f1d11c3-b0a7-4f83-b161-232673bf74f9" path="/var/lib/kubelet/pods/4f1d11c3-b0a7-4f83-b161-232673bf74f9/volumes"
Jul 17 13:03:44.977253 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:44.977219    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:03:45.036644 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.036608    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s"]
Jul 17 13:03:45.036944 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.036930    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="storage-initializer"
Jul 17 13:03:45.037001 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.036946    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="storage-initializer"
Jul 17 13:03:45.037001 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.036956    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="kube-rbac-proxy"
Jul 17 13:03:45.037001 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.036962    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="kube-rbac-proxy"
Jul 17 13:03:45.037001 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.036977    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f1d11c3-b0a7-4f83-b161-232673bf74f9" containerName="storage-initializer"
Jul 17 13:03:45.037001 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.036983    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f1d11c3-b0a7-4f83-b161-232673bf74f9" containerName="storage-initializer"
Jul 17 13:03:45.037001 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.036989    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="752cd404-5062-457f-9223-152507394688" containerName="kube-rbac-proxy"
Jul 17 13:03:45.037001 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.036995    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="752cd404-5062-457f-9223-152507394688" containerName="kube-rbac-proxy"
Jul 17 13:03:45.037001 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.037001    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f1d11c3-b0a7-4f83-b161-232673bf74f9" containerName="kserve-container"
Jul 17 13:03:45.037261 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.037007    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f1d11c3-b0a7-4f83-b161-232673bf74f9" containerName="kserve-container"
Jul 17 13:03:45.037261 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.037012    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f1d11c3-b0a7-4f83-b161-232673bf74f9" containerName="kube-rbac-proxy"
Jul 17 13:03:45.037261 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.037017    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f1d11c3-b0a7-4f83-b161-232673bf74f9" containerName="kube-rbac-proxy"
Jul 17 13:03:45.037261 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.037023    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="agent"
Jul 17 13:03:45.037261 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.037029    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="agent"
Jul 17 13:03:45.037261 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.037035    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="kserve-container"
Jul 17 13:03:45.037261 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.037042    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="kserve-container"
Jul 17 13:03:45.037261 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.037054    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="752cd404-5062-457f-9223-152507394688" containerName="kserve-container"
Jul 17 13:03:45.037261 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.037058    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="752cd404-5062-457f-9223-152507394688" containerName="kserve-container"
Jul 17 13:03:45.037261 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.037103    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f1d11c3-b0a7-4f83-b161-232673bf74f9" containerName="kserve-container"
Jul 17 13:03:45.037261 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.037111    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="kserve-container"
Jul 17 13:03:45.037261 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.037118    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="752cd404-5062-457f-9223-152507394688" containerName="kube-rbac-proxy"
Jul 17 13:03:45.037261 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.037127    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="kube-rbac-proxy"
Jul 17 13:03:45.037261 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.037132    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a1ff06b-005d-4538-bc3b-d73d6d729e44" containerName="agent"
Jul 17 13:03:45.037261 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.037138    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f1d11c3-b0a7-4f83-b161-232673bf74f9" containerName="kube-rbac-proxy"
Jul 17 13:03:45.037261 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.037143    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="752cd404-5062-457f-9223-152507394688" containerName="kserve-container"
Jul 17 13:03:45.061116 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.061050    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s"]
Jul 17 13:03:45.061293 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.061196    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s"
Jul 17 13:03:45.064091 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.064065    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 17 13:03:45.065104 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.065071    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 17 13:03:45.065394 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.065377    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 17 13:03:45.065516 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.065389    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7v8wk\""
Jul 17 13:03:45.065516 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.065376    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 17 13:03:45.065795 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.065771    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 17 13:03:45.065925 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.065777    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 17 13:03:45.192449 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.192420    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xb5s\" (UID: \"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s"
Jul 17 13:03:45.192602 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.192485    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xb5s\" (UID: \"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s"
Jul 17 13:03:45.192602 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.192512    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xb5s\" (UID: \"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s"
Jul 17 13:03:45.192602 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.192534    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrwc9\" (UniqueName: \"kubernetes.io/projected/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-kube-api-access-qrwc9\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xb5s\" (UID: \"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s"
Jul 17 13:03:45.192602 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.192551    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xb5s\" (UID: \"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s"
Jul 17 13:03:45.293280 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.293244    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xb5s\" (UID: \"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s"
Jul 17 13:03:45.293471 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.293338    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xb5s\" (UID: \"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s"
Jul 17 13:03:45.293471 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.293368    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xb5s\" (UID: \"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s"
Jul 17 13:03:45.293471 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.293394    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrwc9\" (UniqueName: \"kubernetes.io/projected/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-kube-api-access-qrwc9\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xb5s\" (UID: \"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s"
Jul 17 13:03:45.293471 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:03:45.293405    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-predictor-serving-cert: secret "isvc-autogluon-v2-predictor-serving-cert" not found
Jul 17 13:03:45.293471 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.293420    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xb5s\" (UID: \"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s"
Jul 17 13:03:45.293805 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:03:45.293522    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-proxy-tls podName:df34c9b5-36c1-47fc-958b-c3d56d9f1b6c nodeName:}" failed. No retries permitted until 2026-07-17 13:03:45.79349971 +0000 UTC m=+2035.373611714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-proxy-tls") pod "isvc-autogluon-v2-predictor-67b849d456-5xb5s" (UID: "df34c9b5-36c1-47fc-958b-c3d56d9f1b6c") : secret "isvc-autogluon-v2-predictor-serving-cert" not found
Jul 17 13:03:45.293805 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.293792    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xb5s\" (UID: \"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s"
Jul 17 13:03:45.294147 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.294126    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xb5s\" (UID: \"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s"
Jul 17 13:03:45.294147 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.294137    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xb5s\" (UID: \"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s"
Jul 17 13:03:45.304347 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.304325    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrwc9\" (UniqueName: \"kubernetes.io/projected/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-kube-api-access-qrwc9\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xb5s\" (UID: \"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s"
Jul 17 13:03:45.423997 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.423961    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:03:45.798199 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.798161    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xb5s\" (UID: \"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s"
Jul 17 13:03:45.800564 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.800540    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-5xb5s\" (UID: \"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s"
Jul 17 13:03:45.972744 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:45.972685    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s"
Jul 17 13:03:46.126236 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:46.126183    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s"]
Jul 17 13:03:46.128504 ip-10-0-137-99 kubenswrapper[2564]: W0717 13:03:46.128470    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf34c9b5_36c1_47fc_958b_c3d56d9f1b6c.slice/crio-0e6a3bcb639e503588b481d7ab541d7580a64210aa24b706dd009a0578a47dc0 WatchSource:0}: Error finding container 0e6a3bcb639e503588b481d7ab541d7580a64210aa24b706dd009a0578a47dc0: Status 404 returned error can't find the container with id 0e6a3bcb639e503588b481d7ab541d7580a64210aa24b706dd009a0578a47dc0
Jul 17 13:03:47.104934 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:47.104884    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s" event={"ID":"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c","Type":"ContainerStarted","Data":"e5c2013d76c156559c67069263aa6a071504101daf9b01b5bda898e8ebc0ac03"}
Jul 17 13:03:47.105135 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:03:47.104957    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s" event={"ID":"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c","Type":"ContainerStarted","Data":"0e6a3bcb639e503588b481d7ab541d7580a64210aa24b706dd009a0578a47dc0"}
Jul 17 13:04:07.187268 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:04:07.187234    2564 generic.go:356] "Generic (PLEG): container finished" podID="df34c9b5-36c1-47fc-958b-c3d56d9f1b6c" containerID="e5c2013d76c156559c67069263aa6a071504101daf9b01b5bda898e8ebc0ac03" exitCode=0
Jul 17 13:04:07.187687 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:04:07.187305    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s" event={"ID":"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c","Type":"ContainerDied","Data":"e5c2013d76c156559c67069263aa6a071504101daf9b01b5bda898e8ebc0ac03"}
Jul 17 13:04:08.194110 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:04:08.194071    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s" event={"ID":"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c","Type":"ContainerStarted","Data":"5b8a61fe0563632abc2c1b0cf02f08d5ef7a5641b36fd9242762901db6a4634d"}
Jul 17 13:04:08.194492 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:04:08.194157    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s" event={"ID":"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c","Type":"ContainerStarted","Data":"837ca944db9b3ff05c8320e1928de5de3531a07b3dff967d19e9120ccb6218d0"}
Jul 17 13:04:08.194492 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:04:08.194373    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s"
Jul 17 13:04:08.215398 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:04:08.215339    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s" podStartSLOduration=3.170969079 podStartE2EDuration="23.215321985s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.044352906s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:04:08.213754654 +0000 UTC m=+2057.793866696" watchObservedRunningTime="2026-07-17 13:04:08.215321985 +0000 UTC m=+2057.795434003"
Jul 17 13:04:09.198545 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:04:09.198510    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s"
Jul 17 13:04:15.209581 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:04:15.209485    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s"
Jul 17 13:04:50.935016 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:04:50.934987    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f958_7ab622df-86d3-4cc3-aba7-123f4bc0c996/ovn-acl-logging/0.log"
Jul 17 13:04:50.938590 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:04:50.938568    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f958_7ab622df-86d3-4cc3-aba7-123f4bc0c996/ovn-acl-logging/0.log"
Jul 17 13:05:41.015797 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:41.015759    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s"
Jul 17 13:05:45.203985 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:45.203883    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s"]
Jul 17 13:05:45.204440 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:45.204166    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s" podUID="df34c9b5-36c1-47fc-958b-c3d56d9f1b6c" containerName="kserve-container" containerID="cri-o://837ca944db9b3ff05c8320e1928de5de3531a07b3dff967d19e9120ccb6218d0" gracePeriod=30
Jul 17 13:05:45.204440 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:45.204250    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s" podUID="df34c9b5-36c1-47fc-958b-c3d56d9f1b6c" containerName="kube-rbac-proxy" containerID="cri-o://5b8a61fe0563632abc2c1b0cf02f08d5ef7a5641b36fd9242762901db6a4634d" gracePeriod=30
Jul 17 13:05:45.271796 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:45.271763    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:05:45.597600 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:45.597551    2564 generic.go:356] "Generic (PLEG): container finished" podID="df34c9b5-36c1-47fc-958b-c3d56d9f1b6c" containerID="5b8a61fe0563632abc2c1b0cf02f08d5ef7a5641b36fd9242762901db6a4634d" exitCode=2
Jul 17 13:05:45.597812 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:45.597630    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s" event={"ID":"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c","Type":"ContainerDied","Data":"5b8a61fe0563632abc2c1b0cf02f08d5ef7a5641b36fd9242762901db6a4634d"}
Jul 17 13:05:46.153842 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.153816    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s"
Jul 17 13:05:46.330986 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.330951    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-kserve-provision-location\") pod \"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c\" (UID: \"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c\") "
Jul 17 13:05:46.331406 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.331069    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-proxy-tls\") pod \"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c\" (UID: \"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c\") "
Jul 17 13:05:46.331406 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.331097    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-cabundle-cert\") pod \"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c\" (UID: \"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c\") "
Jul 17 13:05:46.331406 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.331132    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrwc9\" (UniqueName: \"kubernetes.io/projected/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-kube-api-access-qrwc9\") pod \"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c\" (UID: \"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c\") "
Jul 17 13:05:46.331406 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.331174    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-kube-rbac-proxy-sar-config\") pod \"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c\" (UID: \"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c\") "
Jul 17 13:05:46.331672 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.331648    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "df34c9b5-36c1-47fc-958b-c3d56d9f1b6c" (UID: "df34c9b5-36c1-47fc-958b-c3d56d9f1b6c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:05:46.331672 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.331657    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "df34c9b5-36c1-47fc-958b-c3d56d9f1b6c" (UID: "df34c9b5-36c1-47fc-958b-c3d56d9f1b6c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:05:46.333228 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.333205    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "df34c9b5-36c1-47fc-958b-c3d56d9f1b6c" (UID: "df34c9b5-36c1-47fc-958b-c3d56d9f1b6c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:05:46.333333 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.333315    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-kube-api-access-qrwc9" (OuterVolumeSpecName: "kube-api-access-qrwc9") pod "df34c9b5-36c1-47fc-958b-c3d56d9f1b6c" (UID: "df34c9b5-36c1-47fc-958b-c3d56d9f1b6c"). InnerVolumeSpecName "kube-api-access-qrwc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:05:46.335779 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.335752    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "df34c9b5-36c1-47fc-958b-c3d56d9f1b6c" (UID: "df34c9b5-36c1-47fc-958b-c3d56d9f1b6c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 13:05:46.431810 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.431773    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:05:46.431810 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.431804    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-kserve-provision-location\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:05:46.431810 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.431816    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-proxy-tls\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:05:46.432124 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.431824    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-cabundle-cert\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:05:46.432124 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.431834    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrwc9\" (UniqueName: \"kubernetes.io/projected/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c-kube-api-access-qrwc9\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:05:46.604087 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.604003    2564 generic.go:356] "Generic (PLEG): container finished" podID="df34c9b5-36c1-47fc-958b-c3d56d9f1b6c" containerID="837ca944db9b3ff05c8320e1928de5de3531a07b3dff967d19e9120ccb6218d0" exitCode=0
Jul 17 13:05:46.604087 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.604067    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s"
Jul 17 13:05:46.604087 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.604079    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s" event={"ID":"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c","Type":"ContainerDied","Data":"837ca944db9b3ff05c8320e1928de5de3531a07b3dff967d19e9120ccb6218d0"}
Jul 17 13:05:46.604321 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.604130    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s" event={"ID":"df34c9b5-36c1-47fc-958b-c3d56d9f1b6c","Type":"ContainerDied","Data":"0e6a3bcb639e503588b481d7ab541d7580a64210aa24b706dd009a0578a47dc0"}
Jul 17 13:05:46.604321 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.604156    2564 scope.go:117] "RemoveContainer" containerID="5b8a61fe0563632abc2c1b0cf02f08d5ef7a5641b36fd9242762901db6a4634d"
Jul 17 13:05:46.613122 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.613098    2564 scope.go:117] "RemoveContainer" containerID="837ca944db9b3ff05c8320e1928de5de3531a07b3dff967d19e9120ccb6218d0"
Jul 17 13:05:46.620411 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.620392    2564 scope.go:117] "RemoveContainer" containerID="e5c2013d76c156559c67069263aa6a071504101daf9b01b5bda898e8ebc0ac03"
Jul 17 13:05:46.627728 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.627687    2564 scope.go:117] "RemoveContainer" containerID="5b8a61fe0563632abc2c1b0cf02f08d5ef7a5641b36fd9242762901db6a4634d"
Jul 17 13:05:46.628019 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:05:46.628001    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b8a61fe0563632abc2c1b0cf02f08d5ef7a5641b36fd9242762901db6a4634d\": container with ID starting with 5b8a61fe0563632abc2c1b0cf02f08d5ef7a5641b36fd9242762901db6a4634d not found: ID does not exist" containerID="5b8a61fe0563632abc2c1b0cf02f08d5ef7a5641b36fd9242762901db6a4634d"
Jul 17 13:05:46.628071 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.628029    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8a61fe0563632abc2c1b0cf02f08d5ef7a5641b36fd9242762901db6a4634d"} err="failed to get container status \"5b8a61fe0563632abc2c1b0cf02f08d5ef7a5641b36fd9242762901db6a4634d\": rpc error: code = NotFound desc = could not find container \"5b8a61fe0563632abc2c1b0cf02f08d5ef7a5641b36fd9242762901db6a4634d\": container with ID starting with 5b8a61fe0563632abc2c1b0cf02f08d5ef7a5641b36fd9242762901db6a4634d not found: ID does not exist"
Jul 17 13:05:46.628071 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.628047    2564 scope.go:117] "RemoveContainer" containerID="837ca944db9b3ff05c8320e1928de5de3531a07b3dff967d19e9120ccb6218d0"
Jul 17 13:05:46.628296 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:05:46.628279    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"837ca944db9b3ff05c8320e1928de5de3531a07b3dff967d19e9120ccb6218d0\": container with ID starting with 837ca944db9b3ff05c8320e1928de5de3531a07b3dff967d19e9120ccb6218d0 not found: ID does not exist" containerID="837ca944db9b3ff05c8320e1928de5de3531a07b3dff967d19e9120ccb6218d0"
Jul 17 13:05:46.628333 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.628313    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"837ca944db9b3ff05c8320e1928de5de3531a07b3dff967d19e9120ccb6218d0"} err="failed to get container status \"837ca944db9b3ff05c8320e1928de5de3531a07b3dff967d19e9120ccb6218d0\": rpc error: code = NotFound desc = could not find container \"837ca944db9b3ff05c8320e1928de5de3531a07b3dff967d19e9120ccb6218d0\": container with ID starting with 837ca944db9b3ff05c8320e1928de5de3531a07b3dff967d19e9120ccb6218d0 not found: ID does not exist"
Jul 17 13:05:46.628333 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.628330    2564 scope.go:117] "RemoveContainer" containerID="e5c2013d76c156559c67069263aa6a071504101daf9b01b5bda898e8ebc0ac03"
Jul 17 13:05:46.628556 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:05:46.628539    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5c2013d76c156559c67069263aa6a071504101daf9b01b5bda898e8ebc0ac03\": container with ID starting with e5c2013d76c156559c67069263aa6a071504101daf9b01b5bda898e8ebc0ac03 not found: ID does not exist" containerID="e5c2013d76c156559c67069263aa6a071504101daf9b01b5bda898e8ebc0ac03"
Jul 17 13:05:46.628593 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.628562    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5c2013d76c156559c67069263aa6a071504101daf9b01b5bda898e8ebc0ac03"} err="failed to get container status \"e5c2013d76c156559c67069263aa6a071504101daf9b01b5bda898e8ebc0ac03\": rpc error: code = NotFound desc = could not find container \"e5c2013d76c156559c67069263aa6a071504101daf9b01b5bda898e8ebc0ac03\": container with ID starting with e5c2013d76c156559c67069263aa6a071504101daf9b01b5bda898e8ebc0ac03 not found: ID does not exist"
Jul 17 13:05:46.632222 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.632200    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s"]
Jul 17 13:05:46.637074 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:46.637052    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-5xb5s"]
Jul 17 13:05:47.013711 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:47.013620    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df34c9b5-36c1-47fc-958b-c3d56d9f1b6c" path="/var/lib/kubelet/pods/df34c9b5-36c1-47fc-958b-c3d56d9f1b6c/volumes"
Jul 17 13:05:50.320914 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.320882    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:05:50.375780 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.375725    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh"]
Jul 17 13:05:50.376096 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.376080    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df34c9b5-36c1-47fc-958b-c3d56d9f1b6c" containerName="kserve-container"
Jul 17 13:05:50.376179 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.376099    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="df34c9b5-36c1-47fc-958b-c3d56d9f1b6c" containerName="kserve-container"
Jul 17 13:05:50.376179 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.376119    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df34c9b5-36c1-47fc-958b-c3d56d9f1b6c" containerName="kube-rbac-proxy"
Jul 17 13:05:50.376179 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.376129    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="df34c9b5-36c1-47fc-958b-c3d56d9f1b6c" containerName="kube-rbac-proxy"
Jul 17 13:05:50.376179 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.376155    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df34c9b5-36c1-47fc-958b-c3d56d9f1b6c" containerName="storage-initializer"
Jul 17 13:05:50.376179 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.376164    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="df34c9b5-36c1-47fc-958b-c3d56d9f1b6c" containerName="storage-initializer"
Jul 17 13:05:50.376357 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.376264    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="df34c9b5-36c1-47fc-958b-c3d56d9f1b6c" containerName="kube-rbac-proxy"
Jul 17 13:05:50.376357 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.376281    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="df34c9b5-36c1-47fc-958b-c3d56d9f1b6c" containerName="kserve-container"
Jul 17 13:05:50.381286 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.381262    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh"
Jul 17 13:05:50.383823 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.383796    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 17 13:05:50.384065 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.384047    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 17 13:05:50.384162 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.384095    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 17 13:05:50.384162 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.384106    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 17 13:05:50.384162 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.384112    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 17 13:05:50.385256 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.385240    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7v8wk\""
Jul 17 13:05:50.385325 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.385240    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 17 13:05:50.391633 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.391581    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh"]
Jul 17 13:05:50.565550 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.565500    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1431c054-5578-49e9-8d3d-b99fc97bf144-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh\" (UID: \"1431c054-5578-49e9-8d3d-b99fc97bf144\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh"
Jul 17 13:05:50.565774 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.565568    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1431c054-5578-49e9-8d3d-b99fc97bf144-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh\" (UID: \"1431c054-5578-49e9-8d3d-b99fc97bf144\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh"
Jul 17 13:05:50.565774 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.565671    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1431c054-5578-49e9-8d3d-b99fc97bf144-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh\" (UID: \"1431c054-5578-49e9-8d3d-b99fc97bf144\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh"
Jul 17 13:05:50.565774 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.565730    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1431c054-5578-49e9-8d3d-b99fc97bf144-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh\" (UID: \"1431c054-5578-49e9-8d3d-b99fc97bf144\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh"
Jul 17 13:05:50.565888 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.565819    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v527j\" (UniqueName: \"kubernetes.io/projected/1431c054-5578-49e9-8d3d-b99fc97bf144-kube-api-access-v527j\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh\" (UID: \"1431c054-5578-49e9-8d3d-b99fc97bf144\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh"
Jul 17 13:05:50.666777 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.666656    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1431c054-5578-49e9-8d3d-b99fc97bf144-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh\" (UID: \"1431c054-5578-49e9-8d3d-b99fc97bf144\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh"
Jul 17 13:05:50.666777 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.666717    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1431c054-5578-49e9-8d3d-b99fc97bf144-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh\" (UID: \"1431c054-5578-49e9-8d3d-b99fc97bf144\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh"
Jul 17 13:05:50.666777 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.666747    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v527j\" (UniqueName: \"kubernetes.io/projected/1431c054-5578-49e9-8d3d-b99fc97bf144-kube-api-access-v527j\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh\" (UID: \"1431c054-5578-49e9-8d3d-b99fc97bf144\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh"
Jul 17 13:05:50.667012 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.666817    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1431c054-5578-49e9-8d3d-b99fc97bf144-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh\" (UID: \"1431c054-5578-49e9-8d3d-b99fc97bf144\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh"
Jul 17 13:05:50.667012 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.666844    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1431c054-5578-49e9-8d3d-b99fc97bf144-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh\" (UID: \"1431c054-5578-49e9-8d3d-b99fc97bf144\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh"
Jul 17 13:05:50.667012 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:05:50.666885    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-serving-cert: secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Jul 17 13:05:50.667012 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:05:50.666972    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1431c054-5578-49e9-8d3d-b99fc97bf144-proxy-tls podName:1431c054-5578-49e9-8d3d-b99fc97bf144 nodeName:}" failed. No retries permitted until 2026-07-17 13:05:51.166950593 +0000 UTC m=+2160.747062609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1431c054-5578-49e9-8d3d-b99fc97bf144-proxy-tls") pod "isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh" (UID: "1431c054-5578-49e9-8d3d-b99fc97bf144") : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Jul 17 13:05:50.667249 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.667226    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1431c054-5578-49e9-8d3d-b99fc97bf144-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh\" (UID: \"1431c054-5578-49e9-8d3d-b99fc97bf144\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh"
Jul 17 13:05:50.667428 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.667409    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1431c054-5578-49e9-8d3d-b99fc97bf144-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh\" (UID: \"1431c054-5578-49e9-8d3d-b99fc97bf144\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh"
Jul 17 13:05:50.667475 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.667423    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1431c054-5578-49e9-8d3d-b99fc97bf144-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh\" (UID: \"1431c054-5578-49e9-8d3d-b99fc97bf144\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh"
Jul 17 13:05:50.678312 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.678279    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v527j\" (UniqueName: \"kubernetes.io/projected/1431c054-5578-49e9-8d3d-b99fc97bf144-kube-api-access-v527j\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh\" (UID: \"1431c054-5578-49e9-8d3d-b99fc97bf144\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh"
Jul 17 13:05:50.877713 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:50.877678    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:05:51.169939 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:51.169898    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1431c054-5578-49e9-8d3d-b99fc97bf144-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh\" (UID: \"1431c054-5578-49e9-8d3d-b99fc97bf144\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh"
Jul 17 13:05:51.172378 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:51.172352    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1431c054-5578-49e9-8d3d-b99fc97bf144-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh\" (UID: \"1431c054-5578-49e9-8d3d-b99fc97bf144\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh"
Jul 17 13:05:51.296191 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:51.296160    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7v8wk\""
Jul 17 13:05:51.304452 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:51.304431    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh"
Jul 17 13:05:51.434644 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:51.434607    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh"]
Jul 17 13:05:51.438037 ip-10-0-137-99 kubenswrapper[2564]: W0717 13:05:51.438006    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1431c054_5578_49e9_8d3d_b99fc97bf144.slice/crio-ba5f5246f495ea26f8d568b52bb41b94718ae523a92cfff6285c411634158f45 WatchSource:0}: Error finding container ba5f5246f495ea26f8d568b52bb41b94718ae523a92cfff6285c411634158f45: Status 404 returned error can't find the container with id ba5f5246f495ea26f8d568b52bb41b94718ae523a92cfff6285c411634158f45
Jul 17 13:05:51.442864 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:51.442837    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 17 13:05:51.627019 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:51.626912    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh" event={"ID":"1431c054-5578-49e9-8d3d-b99fc97bf144","Type":"ContainerStarted","Data":"e9c6d5af8c5347ddf6261b8b2590108124cd5063e2b1830ee5c93e388fb767ae"}
Jul 17 13:05:51.627019 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:51.627002    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh" event={"ID":"1431c054-5578-49e9-8d3d-b99fc97bf144","Type":"ContainerStarted","Data":"ba5f5246f495ea26f8d568b52bb41b94718ae523a92cfff6285c411634158f45"}
Jul 17 13:05:56.650432 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:56.650398    2564 generic.go:356] "Generic (PLEG): container finished" podID="1431c054-5578-49e9-8d3d-b99fc97bf144" containerID="e9c6d5af8c5347ddf6261b8b2590108124cd5063e2b1830ee5c93e388fb767ae" exitCode=0
Jul 17 13:05:56.650844 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:56.650475    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh" event={"ID":"1431c054-5578-49e9-8d3d-b99fc97bf144","Type":"ContainerDied","Data":"e9c6d5af8c5347ddf6261b8b2590108124cd5063e2b1830ee5c93e388fb767ae"}
Jul 17 13:05:57.656920 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:57.656883    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh" event={"ID":"1431c054-5578-49e9-8d3d-b99fc97bf144","Type":"ContainerStarted","Data":"9478e45e8a4ea0c395f16a2962ca53abe97d0e955b1c1696a214cea81b3c4250"}
Jul 17 13:05:57.657402 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:57.656958    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh" event={"ID":"1431c054-5578-49e9-8d3d-b99fc97bf144","Type":"ContainerStarted","Data":"c9ec4cd30ae2f10d2c222345da4265682f863f13948a8b3b3ccc3b6bccccd2d8"}
Jul 17 13:05:57.657402 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:57.657211    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh"
Jul 17 13:05:57.657402 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:57.657277    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh"
Jul 17 13:05:57.677912 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:05:57.677847    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh" podStartSLOduration=2.774818517 podStartE2EDuration="7.677825456s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.903006939s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:05:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:05:57.676172443 +0000 UTC m=+2167.256284485" watchObservedRunningTime="2026-07-17 13:05:57.677825456 +0000 UTC m=+2167.257937475"
Jul 17 13:06:03.667572 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:06:03.667541    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh"
Jul 17 13:06:16.600220 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:06:16.600187    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:07:26.013980 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:26.013864    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh"
Jul 17 13:07:30.528948 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:30.528916    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh"]
Jul 17 13:07:30.529333 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:30.529193    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh" podUID="1431c054-5578-49e9-8d3d-b99fc97bf144" containerName="kserve-container" containerID="cri-o://c9ec4cd30ae2f10d2c222345da4265682f863f13948a8b3b3ccc3b6bccccd2d8" gracePeriod=30
Jul 17 13:07:30.529333 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:30.529194    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh" podUID="1431c054-5578-49e9-8d3d-b99fc97bf144" containerName="kube-rbac-proxy" containerID="cri-o://9478e45e8a4ea0c395f16a2962ca53abe97d0e955b1c1696a214cea81b3c4250" gracePeriod=30
Jul 17 13:07:30.557858 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:30.557834    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:07:31.051868 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:31.051830    2564 generic.go:356] "Generic (PLEG): container finished" podID="1431c054-5578-49e9-8d3d-b99fc97bf144" containerID="9478e45e8a4ea0c395f16a2962ca53abe97d0e955b1c1696a214cea81b3c4250" exitCode=2
Jul 17 13:07:31.052043 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:31.051904    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh" event={"ID":"1431c054-5578-49e9-8d3d-b99fc97bf144","Type":"ContainerDied","Data":"9478e45e8a4ea0c395f16a2962ca53abe97d0e955b1c1696a214cea81b3c4250"}
Jul 17 13:07:31.377661 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:31.377634    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh"
Jul 17 13:07:31.440224 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:31.440181    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v527j\" (UniqueName: \"kubernetes.io/projected/1431c054-5578-49e9-8d3d-b99fc97bf144-kube-api-access-v527j\") pod \"1431c054-5578-49e9-8d3d-b99fc97bf144\" (UID: \"1431c054-5578-49e9-8d3d-b99fc97bf144\") "
Jul 17 13:07:31.440393 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:31.440238    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1431c054-5578-49e9-8d3d-b99fc97bf144-kube-rbac-proxy-sar-config\") pod \"1431c054-5578-49e9-8d3d-b99fc97bf144\" (UID: \"1431c054-5578-49e9-8d3d-b99fc97bf144\") "
Jul 17 13:07:31.440393 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:31.440279    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1431c054-5578-49e9-8d3d-b99fc97bf144-kserve-provision-location\") pod \"1431c054-5578-49e9-8d3d-b99fc97bf144\" (UID: \"1431c054-5578-49e9-8d3d-b99fc97bf144\") "
Jul 17 13:07:31.440393 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:31.440302    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1431c054-5578-49e9-8d3d-b99fc97bf144-cabundle-cert\") pod \"1431c054-5578-49e9-8d3d-b99fc97bf144\" (UID: \"1431c054-5578-49e9-8d3d-b99fc97bf144\") "
Jul 17 13:07:31.440393 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:31.440321    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1431c054-5578-49e9-8d3d-b99fc97bf144-proxy-tls\") pod \"1431c054-5578-49e9-8d3d-b99fc97bf144\" (UID: \"1431c054-5578-49e9-8d3d-b99fc97bf144\") "
Jul 17 13:07:31.440650 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:31.440625    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1431c054-5578-49e9-8d3d-b99fc97bf144-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1431c054-5578-49e9-8d3d-b99fc97bf144" (UID: "1431c054-5578-49e9-8d3d-b99fc97bf144"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:07:31.440733 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:31.440686    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1431c054-5578-49e9-8d3d-b99fc97bf144-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1431c054-5578-49e9-8d3d-b99fc97bf144" (UID: "1431c054-5578-49e9-8d3d-b99fc97bf144"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:07:31.442501 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:31.442479    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1431c054-5578-49e9-8d3d-b99fc97bf144-kube-api-access-v527j" (OuterVolumeSpecName: "kube-api-access-v527j") pod "1431c054-5578-49e9-8d3d-b99fc97bf144" (UID: "1431c054-5578-49e9-8d3d-b99fc97bf144"). InnerVolumeSpecName "kube-api-access-v527j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:07:31.442600 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:31.442515    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1431c054-5578-49e9-8d3d-b99fc97bf144-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1431c054-5578-49e9-8d3d-b99fc97bf144" (UID: "1431c054-5578-49e9-8d3d-b99fc97bf144"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:07:31.444851 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:31.444827    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1431c054-5578-49e9-8d3d-b99fc97bf144-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1431c054-5578-49e9-8d3d-b99fc97bf144" (UID: "1431c054-5578-49e9-8d3d-b99fc97bf144"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 13:07:31.541353 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:31.541321    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1431c054-5578-49e9-8d3d-b99fc97bf144-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:07:31.541353 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:31.541351    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1431c054-5578-49e9-8d3d-b99fc97bf144-kserve-provision-location\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:07:31.541353 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:31.541361    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1431c054-5578-49e9-8d3d-b99fc97bf144-cabundle-cert\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:07:31.541353 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:31.541371    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1431c054-5578-49e9-8d3d-b99fc97bf144-proxy-tls\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:07:31.541861 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:31.541379    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v527j\" (UniqueName: \"kubernetes.io/projected/1431c054-5578-49e9-8d3d-b99fc97bf144-kube-api-access-v527j\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:07:32.057917 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:32.057884    2564 generic.go:356] "Generic (PLEG): container finished" podID="1431c054-5578-49e9-8d3d-b99fc97bf144" containerID="c9ec4cd30ae2f10d2c222345da4265682f863f13948a8b3b3ccc3b6bccccd2d8" exitCode=0
Jul 17 13:07:32.058067 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:32.057943    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh"
Jul 17 13:07:32.058067 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:32.057964    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh" event={"ID":"1431c054-5578-49e9-8d3d-b99fc97bf144","Type":"ContainerDied","Data":"c9ec4cd30ae2f10d2c222345da4265682f863f13948a8b3b3ccc3b6bccccd2d8"}
Jul 17 13:07:32.058067 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:32.058026    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh" event={"ID":"1431c054-5578-49e9-8d3d-b99fc97bf144","Type":"ContainerDied","Data":"ba5f5246f495ea26f8d568b52bb41b94718ae523a92cfff6285c411634158f45"}
Jul 17 13:07:32.058067 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:32.058055    2564 scope.go:117] "RemoveContainer" containerID="9478e45e8a4ea0c395f16a2962ca53abe97d0e955b1c1696a214cea81b3c4250"
Jul 17 13:07:32.066928 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:32.066901    2564 scope.go:117] "RemoveContainer" containerID="c9ec4cd30ae2f10d2c222345da4265682f863f13948a8b3b3ccc3b6bccccd2d8"
Jul 17 13:07:32.074616 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:32.074600    2564 scope.go:117] "RemoveContainer" containerID="e9c6d5af8c5347ddf6261b8b2590108124cd5063e2b1830ee5c93e388fb767ae"
Jul 17 13:07:32.081814 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:32.081793    2564 scope.go:117] "RemoveContainer" containerID="9478e45e8a4ea0c395f16a2962ca53abe97d0e955b1c1696a214cea81b3c4250"
Jul 17 13:07:32.082071 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:07:32.082054    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9478e45e8a4ea0c395f16a2962ca53abe97d0e955b1c1696a214cea81b3c4250\": container with ID starting with 9478e45e8a4ea0c395f16a2962ca53abe97d0e955b1c1696a214cea81b3c4250 not found: ID does not exist" containerID="9478e45e8a4ea0c395f16a2962ca53abe97d0e955b1c1696a214cea81b3c4250"
Jul 17 13:07:32.082108 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:32.082081    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9478e45e8a4ea0c395f16a2962ca53abe97d0e955b1c1696a214cea81b3c4250"} err="failed to get container status \"9478e45e8a4ea0c395f16a2962ca53abe97d0e955b1c1696a214cea81b3c4250\": rpc error: code = NotFound desc = could not find container \"9478e45e8a4ea0c395f16a2962ca53abe97d0e955b1c1696a214cea81b3c4250\": container with ID starting with 9478e45e8a4ea0c395f16a2962ca53abe97d0e955b1c1696a214cea81b3c4250 not found: ID does not exist"
Jul 17 13:07:32.082108 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:32.082102    2564 scope.go:117] "RemoveContainer" containerID="c9ec4cd30ae2f10d2c222345da4265682f863f13948a8b3b3ccc3b6bccccd2d8"
Jul 17 13:07:32.082327 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:07:32.082311    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9ec4cd30ae2f10d2c222345da4265682f863f13948a8b3b3ccc3b6bccccd2d8\": container with ID starting with c9ec4cd30ae2f10d2c222345da4265682f863f13948a8b3b3ccc3b6bccccd2d8 not found: ID does not exist" containerID="c9ec4cd30ae2f10d2c222345da4265682f863f13948a8b3b3ccc3b6bccccd2d8"
Jul 17 13:07:32.082363 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:32.082332    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9ec4cd30ae2f10d2c222345da4265682f863f13948a8b3b3ccc3b6bccccd2d8"} err="failed to get container status \"c9ec4cd30ae2f10d2c222345da4265682f863f13948a8b3b3ccc3b6bccccd2d8\": rpc error: code = NotFound desc = could not find container \"c9ec4cd30ae2f10d2c222345da4265682f863f13948a8b3b3ccc3b6bccccd2d8\": container with ID starting with c9ec4cd30ae2f10d2c222345da4265682f863f13948a8b3b3ccc3b6bccccd2d8 not found: ID does not exist"
Jul 17 13:07:32.082363 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:32.082346    2564 scope.go:117] "RemoveContainer" containerID="e9c6d5af8c5347ddf6261b8b2590108124cd5063e2b1830ee5c93e388fb767ae"
Jul 17 13:07:32.082586 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:07:32.082562    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9c6d5af8c5347ddf6261b8b2590108124cd5063e2b1830ee5c93e388fb767ae\": container with ID starting with e9c6d5af8c5347ddf6261b8b2590108124cd5063e2b1830ee5c93e388fb767ae not found: ID does not exist" containerID="e9c6d5af8c5347ddf6261b8b2590108124cd5063e2b1830ee5c93e388fb767ae"
Jul 17 13:07:32.082682 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:32.082592    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9c6d5af8c5347ddf6261b8b2590108124cd5063e2b1830ee5c93e388fb767ae"} err="failed to get container status \"e9c6d5af8c5347ddf6261b8b2590108124cd5063e2b1830ee5c93e388fb767ae\": rpc error: code = NotFound desc = could not find container \"e9c6d5af8c5347ddf6261b8b2590108124cd5063e2b1830ee5c93e388fb767ae\": container with ID starting with e9c6d5af8c5347ddf6261b8b2590108124cd5063e2b1830ee5c93e388fb767ae not found: ID does not exist"
Jul 17 13:07:32.091934 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:32.091907    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh"]
Jul 17 13:07:32.102260 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:32.102235    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-7lhsh"]
Jul 17 13:07:33.013962 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:33.013931    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1431c054-5578-49e9-8d3d-b99fc97bf144" path="/var/lib/kubelet/pods/1431c054-5578-49e9-8d3d-b99fc97bf144/volumes"
Jul 17 13:07:35.656719 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.656582    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:07:35.728625 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.728589    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj"]
Jul 17 13:07:35.728928 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.728916    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1431c054-5578-49e9-8d3d-b99fc97bf144" containerName="kube-rbac-proxy"
Jul 17 13:07:35.728984 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.728930    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1431c054-5578-49e9-8d3d-b99fc97bf144" containerName="kube-rbac-proxy"
Jul 17 13:07:35.728984 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.728944    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1431c054-5578-49e9-8d3d-b99fc97bf144" containerName="storage-initializer"
Jul 17 13:07:35.728984 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.728952    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1431c054-5578-49e9-8d3d-b99fc97bf144" containerName="storage-initializer"
Jul 17 13:07:35.728984 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.728959    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1431c054-5578-49e9-8d3d-b99fc97bf144" containerName="kserve-container"
Jul 17 13:07:35.728984 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.728965    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1431c054-5578-49e9-8d3d-b99fc97bf144" containerName="kserve-container"
Jul 17 13:07:35.729166 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.729017    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1431c054-5578-49e9-8d3d-b99fc97bf144" containerName="kube-rbac-proxy"
Jul 17 13:07:35.729166 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.729027    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1431c054-5578-49e9-8d3d-b99fc97bf144" containerName="kserve-container"
Jul 17 13:07:35.733961 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.733939    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj"
Jul 17 13:07:35.736647 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.736624    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7v8wk\""
Jul 17 13:07:35.736814 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.736645    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 17 13:07:35.736814 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.736627    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 17 13:07:35.736896 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.736844    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 17 13:07:35.737796 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.737777    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 17 13:07:35.737912 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.737798    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 17 13:07:35.737912 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.737784    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 17 13:07:35.742576 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.742554    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj"]
Jul 17 13:07:35.778393 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.778361    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20c78948-621c-493c-bf8c-117cac507dc8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj\" (UID: \"20c78948-621c-493c-bf8c-117cac507dc8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj"
Jul 17 13:07:35.778575 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.778400    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzcd8\" (UniqueName: \"kubernetes.io/projected/20c78948-621c-493c-bf8c-117cac507dc8-kube-api-access-wzcd8\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj\" (UID: \"20c78948-621c-493c-bf8c-117cac507dc8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj"
Jul 17 13:07:35.778575 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.778427    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20c78948-621c-493c-bf8c-117cac507dc8-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj\" (UID: \"20c78948-621c-493c-bf8c-117cac507dc8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj"
Jul 17 13:07:35.778575 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.778457    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20c78948-621c-493c-bf8c-117cac507dc8-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj\" (UID: \"20c78948-621c-493c-bf8c-117cac507dc8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj"
Jul 17 13:07:35.778575 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.778524    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20c78948-621c-493c-bf8c-117cac507dc8-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj\" (UID: \"20c78948-621c-493c-bf8c-117cac507dc8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj"
Jul 17 13:07:35.879572 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.879528    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20c78948-621c-493c-bf8c-117cac507dc8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj\" (UID: \"20c78948-621c-493c-bf8c-117cac507dc8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj"
Jul 17 13:07:35.879572 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.879576    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzcd8\" (UniqueName: \"kubernetes.io/projected/20c78948-621c-493c-bf8c-117cac507dc8-kube-api-access-wzcd8\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj\" (UID: \"20c78948-621c-493c-bf8c-117cac507dc8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj"
Jul 17 13:07:35.879907 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.879614    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20c78948-621c-493c-bf8c-117cac507dc8-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj\" (UID: \"20c78948-621c-493c-bf8c-117cac507dc8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj"
Jul 17 13:07:35.879907 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.879648    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20c78948-621c-493c-bf8c-117cac507dc8-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj\" (UID: \"20c78948-621c-493c-bf8c-117cac507dc8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj"
Jul 17 13:07:35.879907 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.879861    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20c78948-621c-493c-bf8c-117cac507dc8-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj\" (UID: \"20c78948-621c-493c-bf8c-117cac507dc8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj"
Jul 17 13:07:35.880064 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:07:35.879981    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-serving-cert: secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Jul 17 13:07:35.880064 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:07:35.880060    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20c78948-621c-493c-bf8c-117cac507dc8-proxy-tls podName:20c78948-621c-493c-bf8c-117cac507dc8 nodeName:}" failed. No retries permitted until 2026-07-17 13:07:36.38003769 +0000 UTC m=+2265.960149702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/20c78948-621c-493c-bf8c-117cac507dc8-proxy-tls") pod "isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj" (UID: "20c78948-621c-493c-bf8c-117cac507dc8") : secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Jul 17 13:07:35.880175 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.880107    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20c78948-621c-493c-bf8c-117cac507dc8-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj\" (UID: \"20c78948-621c-493c-bf8c-117cac507dc8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj"
Jul 17 13:07:35.880362 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.880344    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20c78948-621c-493c-bf8c-117cac507dc8-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj\" (UID: \"20c78948-621c-493c-bf8c-117cac507dc8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj"
Jul 17 13:07:35.880399 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.880360    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20c78948-621c-493c-bf8c-117cac507dc8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj\" (UID: \"20c78948-621c-493c-bf8c-117cac507dc8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj"
Jul 17 13:07:35.889203 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:35.889181    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzcd8\" (UniqueName: \"kubernetes.io/projected/20c78948-621c-493c-bf8c-117cac507dc8-kube-api-access-wzcd8\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj\" (UID: \"20c78948-621c-493c-bf8c-117cac507dc8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj"
Jul 17 13:07:36.105142 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:36.105107    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:07:36.384421 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:36.384322    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20c78948-621c-493c-bf8c-117cac507dc8-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj\" (UID: \"20c78948-621c-493c-bf8c-117cac507dc8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj"
Jul 17 13:07:36.386780 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:36.386761    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20c78948-621c-493c-bf8c-117cac507dc8-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj\" (UID: \"20c78948-621c-493c-bf8c-117cac507dc8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj"
Jul 17 13:07:36.646770 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:36.646647    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj"
Jul 17 13:07:36.782671 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:36.782638    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj"]
Jul 17 13:07:36.786458 ip-10-0-137-99 kubenswrapper[2564]: W0717 13:07:36.786425    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20c78948_621c_493c_bf8c_117cac507dc8.slice/crio-c8ad2b5516abd6bda3be4cb2e84ebc6c1af1f7a9bb35fe8f6ec60575dcb29cfd WatchSource:0}: Error finding container c8ad2b5516abd6bda3be4cb2e84ebc6c1af1f7a9bb35fe8f6ec60575dcb29cfd: Status 404 returned error can't find the container with id c8ad2b5516abd6bda3be4cb2e84ebc6c1af1f7a9bb35fe8f6ec60575dcb29cfd
Jul 17 13:07:37.083484 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:37.083439    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj" event={"ID":"20c78948-621c-493c-bf8c-117cac507dc8","Type":"ContainerStarted","Data":"b007906ebb23a9f930ef93dfc4131c50d84ad4d32062be1dbc547951f102f430"}
Jul 17 13:07:37.083662 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:37.083524    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj" event={"ID":"20c78948-621c-493c-bf8c-117cac507dc8","Type":"ContainerStarted","Data":"c8ad2b5516abd6bda3be4cb2e84ebc6c1af1f7a9bb35fe8f6ec60575dcb29cfd"}
Jul 17 13:07:42.105263 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:42.105228    2564 generic.go:356] "Generic (PLEG): container finished" podID="20c78948-621c-493c-bf8c-117cac507dc8" containerID="b007906ebb23a9f930ef93dfc4131c50d84ad4d32062be1dbc547951f102f430" exitCode=0
Jul 17 13:07:42.105681 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:42.105302    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj" event={"ID":"20c78948-621c-493c-bf8c-117cac507dc8","Type":"ContainerDied","Data":"b007906ebb23a9f930ef93dfc4131c50d84ad4d32062be1dbc547951f102f430"}
Jul 17 13:07:43.111773 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:43.111736    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj" event={"ID":"20c78948-621c-493c-bf8c-117cac507dc8","Type":"ContainerStarted","Data":"7b9201115d830d00d0231ebbf73b10a1bcd2f20cf195ab1386077776024b1e20"}
Jul 17 13:07:43.112166 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:43.111803    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj" event={"ID":"20c78948-621c-493c-bf8c-117cac507dc8","Type":"ContainerStarted","Data":"920487191e32b1edf44c452cc5ce685c81871fcad6b63ad82baf5af620af09ce"}
Jul 17 13:07:43.112166 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:43.112009    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj"
Jul 17 13:07:43.132581 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:43.132528    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj" podStartSLOduration=3.600154968 podStartE2EDuration="8.132509311s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.532354343s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:07:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:07:43.131836279 +0000 UTC m=+2272.711948316" watchObservedRunningTime="2026-07-17 13:07:43.132509311 +0000 UTC m=+2272.712621335"
Jul 17 13:07:44.116673 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:44.116646    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj"
Jul 17 13:07:50.126407 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:07:50.126366    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj"
Jul 17 13:08:54.013340 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:08:54.013239    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:09:12.013870 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:12.013832    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj"
Jul 17 13:09:15.827766 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:15.827732    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj"]
Jul 17 13:09:15.828180 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:15.827956    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj" podUID="20c78948-621c-493c-bf8c-117cac507dc8" containerName="kserve-container" containerID="cri-o://920487191e32b1edf44c452cc5ce685c81871fcad6b63ad82baf5af620af09ce" gracePeriod=30
Jul 17 13:09:15.828180 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:15.828058    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj" podUID="20c78948-621c-493c-bf8c-117cac507dc8" containerName="kube-rbac-proxy" containerID="cri-o://7b9201115d830d00d0231ebbf73b10a1bcd2f20cf195ab1386077776024b1e20" gracePeriod=30
Jul 17 13:09:15.849321 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:15.849292    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:09:16.507003 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:16.506967    2564 generic.go:356] "Generic (PLEG): container finished" podID="20c78948-621c-493c-bf8c-117cac507dc8" containerID="7b9201115d830d00d0231ebbf73b10a1bcd2f20cf195ab1386077776024b1e20" exitCode=2
Jul 17 13:09:16.507178 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:16.507042    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj" event={"ID":"20c78948-621c-493c-bf8c-117cac507dc8","Type":"ContainerDied","Data":"7b9201115d830d00d0231ebbf73b10a1bcd2f20cf195ab1386077776024b1e20"}
Jul 17 13:09:16.776120 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:16.776090    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj"
Jul 17 13:09:16.849407 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:16.849372    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzcd8\" (UniqueName: \"kubernetes.io/projected/20c78948-621c-493c-bf8c-117cac507dc8-kube-api-access-wzcd8\") pod \"20c78948-621c-493c-bf8c-117cac507dc8\" (UID: \"20c78948-621c-493c-bf8c-117cac507dc8\") "
Jul 17 13:09:16.849970 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:16.849419    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20c78948-621c-493c-bf8c-117cac507dc8-proxy-tls\") pod \"20c78948-621c-493c-bf8c-117cac507dc8\" (UID: \"20c78948-621c-493c-bf8c-117cac507dc8\") "
Jul 17 13:09:16.849970 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:16.849441    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20c78948-621c-493c-bf8c-117cac507dc8-kube-rbac-proxy-sar-config\") pod \"20c78948-621c-493c-bf8c-117cac507dc8\" (UID: \"20c78948-621c-493c-bf8c-117cac507dc8\") "
Jul 17 13:09:16.849970 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:16.849464    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20c78948-621c-493c-bf8c-117cac507dc8-kserve-provision-location\") pod \"20c78948-621c-493c-bf8c-117cac507dc8\" (UID: \"20c78948-621c-493c-bf8c-117cac507dc8\") "
Jul 17 13:09:16.849970 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:16.849486    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20c78948-621c-493c-bf8c-117cac507dc8-cabundle-cert\") pod \"20c78948-621c-493c-bf8c-117cac507dc8\" (UID: \"20c78948-621c-493c-bf8c-117cac507dc8\") "
Jul 17 13:09:16.849970 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:16.849914    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c78948-621c-493c-bf8c-117cac507dc8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "20c78948-621c-493c-bf8c-117cac507dc8" (UID: "20c78948-621c-493c-bf8c-117cac507dc8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:09:16.849970 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:16.849922    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c78948-621c-493c-bf8c-117cac507dc8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "20c78948-621c-493c-bf8c-117cac507dc8" (UID: "20c78948-621c-493c-bf8c-117cac507dc8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:09:16.851617 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:16.851589    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c78948-621c-493c-bf8c-117cac507dc8-kube-api-access-wzcd8" (OuterVolumeSpecName: "kube-api-access-wzcd8") pod "20c78948-621c-493c-bf8c-117cac507dc8" (UID: "20c78948-621c-493c-bf8c-117cac507dc8"). InnerVolumeSpecName "kube-api-access-wzcd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:09:16.852153 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:16.852134    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20c78948-621c-493c-bf8c-117cac507dc8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "20c78948-621c-493c-bf8c-117cac507dc8" (UID: "20c78948-621c-493c-bf8c-117cac507dc8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:09:16.855149 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:16.855114    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20c78948-621c-493c-bf8c-117cac507dc8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "20c78948-621c-493c-bf8c-117cac507dc8" (UID: "20c78948-621c-493c-bf8c-117cac507dc8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 13:09:16.950296 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:16.950254    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzcd8\" (UniqueName: \"kubernetes.io/projected/20c78948-621c-493c-bf8c-117cac507dc8-kube-api-access-wzcd8\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:09:16.950296 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:16.950286    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20c78948-621c-493c-bf8c-117cac507dc8-proxy-tls\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:09:16.950296 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:16.950297    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20c78948-621c-493c-bf8c-117cac507dc8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:09:16.950296 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:16.950305    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20c78948-621c-493c-bf8c-117cac507dc8-kserve-provision-location\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:09:16.950576 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:16.950314    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20c78948-621c-493c-bf8c-117cac507dc8-cabundle-cert\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:09:17.513598 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:17.513561    2564 generic.go:356] "Generic (PLEG): container finished" podID="20c78948-621c-493c-bf8c-117cac507dc8" containerID="920487191e32b1edf44c452cc5ce685c81871fcad6b63ad82baf5af620af09ce" exitCode=0
Jul 17 13:09:17.513813 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:17.513612    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj" event={"ID":"20c78948-621c-493c-bf8c-117cac507dc8","Type":"ContainerDied","Data":"920487191e32b1edf44c452cc5ce685c81871fcad6b63ad82baf5af620af09ce"}
Jul 17 13:09:17.513813 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:17.513616    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj"
Jul 17 13:09:17.513813 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:17.513661    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj" event={"ID":"20c78948-621c-493c-bf8c-117cac507dc8","Type":"ContainerDied","Data":"c8ad2b5516abd6bda3be4cb2e84ebc6c1af1f7a9bb35fe8f6ec60575dcb29cfd"}
Jul 17 13:09:17.513813 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:17.513687    2564 scope.go:117] "RemoveContainer" containerID="7b9201115d830d00d0231ebbf73b10a1bcd2f20cf195ab1386077776024b1e20"
Jul 17 13:09:17.522024 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:17.522005    2564 scope.go:117] "RemoveContainer" containerID="920487191e32b1edf44c452cc5ce685c81871fcad6b63ad82baf5af620af09ce"
Jul 17 13:09:17.529564 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:17.529542    2564 scope.go:117] "RemoveContainer" containerID="b007906ebb23a9f930ef93dfc4131c50d84ad4d32062be1dbc547951f102f430"
Jul 17 13:09:17.534301 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:17.534275    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj"]
Jul 17 13:09:17.537954 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:17.537927    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-mmlgj"]
Jul 17 13:09:17.538675 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:17.538653    2564 scope.go:117] "RemoveContainer" containerID="7b9201115d830d00d0231ebbf73b10a1bcd2f20cf195ab1386077776024b1e20"
Jul 17 13:09:17.539071 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:09:17.539054    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b9201115d830d00d0231ebbf73b10a1bcd2f20cf195ab1386077776024b1e20\": container with ID starting with 7b9201115d830d00d0231ebbf73b10a1bcd2f20cf195ab1386077776024b1e20 not found: ID does not exist" containerID="7b9201115d830d00d0231ebbf73b10a1bcd2f20cf195ab1386077776024b1e20"
Jul 17 13:09:17.539144 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:17.539080    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b9201115d830d00d0231ebbf73b10a1bcd2f20cf195ab1386077776024b1e20"} err="failed to get container status \"7b9201115d830d00d0231ebbf73b10a1bcd2f20cf195ab1386077776024b1e20\": rpc error: code = NotFound desc = could not find container \"7b9201115d830d00d0231ebbf73b10a1bcd2f20cf195ab1386077776024b1e20\": container with ID starting with 7b9201115d830d00d0231ebbf73b10a1bcd2f20cf195ab1386077776024b1e20 not found: ID does not exist"
Jul 17 13:09:17.539144 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:17.539102    2564 scope.go:117] "RemoveContainer" containerID="920487191e32b1edf44c452cc5ce685c81871fcad6b63ad82baf5af620af09ce"
Jul 17 13:09:17.539389 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:09:17.539372    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"920487191e32b1edf44c452cc5ce685c81871fcad6b63ad82baf5af620af09ce\": container with ID starting with 920487191e32b1edf44c452cc5ce685c81871fcad6b63ad82baf5af620af09ce not found: ID does not exist" containerID="920487191e32b1edf44c452cc5ce685c81871fcad6b63ad82baf5af620af09ce"
Jul 17 13:09:17.539443 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:17.539393    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"920487191e32b1edf44c452cc5ce685c81871fcad6b63ad82baf5af620af09ce"} err="failed to get container status \"920487191e32b1edf44c452cc5ce685c81871fcad6b63ad82baf5af620af09ce\": rpc error: code = NotFound desc = could not find container \"920487191e32b1edf44c452cc5ce685c81871fcad6b63ad82baf5af620af09ce\": container with ID starting with 920487191e32b1edf44c452cc5ce685c81871fcad6b63ad82baf5af620af09ce not found: ID does not exist"
Jul 17 13:09:17.539443 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:17.539409    2564 scope.go:117] "RemoveContainer" containerID="b007906ebb23a9f930ef93dfc4131c50d84ad4d32062be1dbc547951f102f430"
Jul 17 13:09:17.539644 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:09:17.539628    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b007906ebb23a9f930ef93dfc4131c50d84ad4d32062be1dbc547951f102f430\": container with ID starting with b007906ebb23a9f930ef93dfc4131c50d84ad4d32062be1dbc547951f102f430 not found: ID does not exist" containerID="b007906ebb23a9f930ef93dfc4131c50d84ad4d32062be1dbc547951f102f430"
Jul 17 13:09:17.539682 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:17.539649    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b007906ebb23a9f930ef93dfc4131c50d84ad4d32062be1dbc547951f102f430"} err="failed to get container status \"b007906ebb23a9f930ef93dfc4131c50d84ad4d32062be1dbc547951f102f430\": rpc error: code = NotFound desc = could not find container \"b007906ebb23a9f930ef93dfc4131c50d84ad4d32062be1dbc547951f102f430\": container with ID starting with b007906ebb23a9f930ef93dfc4131c50d84ad4d32062be1dbc547951f102f430 not found: ID does not exist"
Jul 17 13:09:19.013911 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:19.013876    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20c78948-621c-493c-bf8c-117cac507dc8" path="/var/lib/kubelet/pods/20c78948-621c-493c-bf8c-117cac507dc8/volumes"
Jul 17 13:09:20.944384 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:20.944349    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:09:21.007603 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.007562    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6"]
Jul 17 13:09:21.008004 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.007985    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20c78948-621c-493c-bf8c-117cac507dc8" containerName="kube-rbac-proxy"
Jul 17 13:09:21.008101 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.008007    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="20c78948-621c-493c-bf8c-117cac507dc8" containerName="kube-rbac-proxy"
Jul 17 13:09:21.008101 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.008024    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20c78948-621c-493c-bf8c-117cac507dc8" containerName="storage-initializer"
Jul 17 13:09:21.008101 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.008032    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="20c78948-621c-493c-bf8c-117cac507dc8" containerName="storage-initializer"
Jul 17 13:09:21.008101 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.008043    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20c78948-621c-493c-bf8c-117cac507dc8" containerName="kserve-container"
Jul 17 13:09:21.008101 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.008053    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="20c78948-621c-493c-bf8c-117cac507dc8" containerName="kserve-container"
Jul 17 13:09:21.008349 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.008141    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="20c78948-621c-493c-bf8c-117cac507dc8" containerName="kserve-container"
Jul 17 13:09:21.008349 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.008153    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="20c78948-621c-493c-bf8c-117cac507dc8" containerName="kube-rbac-proxy"
Jul 17 13:09:21.013220 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.013190    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6"
Jul 17 13:09:21.017940 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.017919    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 17 13:09:21.018053 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.017943    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 17 13:09:21.018393 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.018372    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 17 13:09:21.018499 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.018389    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 17 13:09:21.018499 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.018397    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 17 13:09:21.018890 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.018862    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7v8wk\""
Jul 17 13:09:21.018986 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.018863    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 17 13:09:21.023247 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.023225    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6"]
Jul 17 13:09:21.076282 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.076247    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:09:21.085419 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.085393    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/faa9678f-5bc2-479c-b6cb-02097a94b089-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6\" (UID: \"faa9678f-5bc2-479c-b6cb-02097a94b089\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6"
Jul 17 13:09:21.085548 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.085426    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/faa9678f-5bc2-479c-b6cb-02097a94b089-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6\" (UID: \"faa9678f-5bc2-479c-b6cb-02097a94b089\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6"
Jul 17 13:09:21.085548 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.085453    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/faa9678f-5bc2-479c-b6cb-02097a94b089-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6\" (UID: \"faa9678f-5bc2-479c-b6cb-02097a94b089\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6"
Jul 17 13:09:21.085548 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.085479    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56wnj\" (UniqueName: \"kubernetes.io/projected/faa9678f-5bc2-479c-b6cb-02097a94b089-kube-api-access-56wnj\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6\" (UID: \"faa9678f-5bc2-479c-b6cb-02097a94b089\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6"
Jul 17 13:09:21.085548 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.085527    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/faa9678f-5bc2-479c-b6cb-02097a94b089-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6\" (UID: \"faa9678f-5bc2-479c-b6cb-02097a94b089\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6"
Jul 17 13:09:21.186597 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.186559    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/faa9678f-5bc2-479c-b6cb-02097a94b089-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6\" (UID: \"faa9678f-5bc2-479c-b6cb-02097a94b089\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6"
Jul 17 13:09:21.186840 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.186659    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/faa9678f-5bc2-479c-b6cb-02097a94b089-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6\" (UID: \"faa9678f-5bc2-479c-b6cb-02097a94b089\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6"
Jul 17 13:09:21.186840 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.186686    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/faa9678f-5bc2-479c-b6cb-02097a94b089-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6\" (UID: \"faa9678f-5bc2-479c-b6cb-02097a94b089\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6"
Jul 17 13:09:21.186840 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.186748    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/faa9678f-5bc2-479c-b6cb-02097a94b089-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6\" (UID: \"faa9678f-5bc2-479c-b6cb-02097a94b089\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6"
Jul 17 13:09:21.186840 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.186779    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56wnj\" (UniqueName: \"kubernetes.io/projected/faa9678f-5bc2-479c-b6cb-02097a94b089-kube-api-access-56wnj\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6\" (UID: \"faa9678f-5bc2-479c-b6cb-02097a94b089\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6"
Jul 17 13:09:21.187122 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.187096    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/faa9678f-5bc2-479c-b6cb-02097a94b089-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6\" (UID: \"faa9678f-5bc2-479c-b6cb-02097a94b089\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6"
Jul 17 13:09:21.187412 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.187385    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/faa9678f-5bc2-479c-b6cb-02097a94b089-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6\" (UID: \"faa9678f-5bc2-479c-b6cb-02097a94b089\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6"
Jul 17 13:09:21.187412 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.187404    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/faa9678f-5bc2-479c-b6cb-02097a94b089-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6\" (UID: \"faa9678f-5bc2-479c-b6cb-02097a94b089\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6"
Jul 17 13:09:21.189183 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.189152    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/faa9678f-5bc2-479c-b6cb-02097a94b089-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6\" (UID: \"faa9678f-5bc2-479c-b6cb-02097a94b089\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6"
Jul 17 13:09:21.196056 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.196004    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56wnj\" (UniqueName: \"kubernetes.io/projected/faa9678f-5bc2-479c-b6cb-02097a94b089-kube-api-access-56wnj\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6\" (UID: \"faa9678f-5bc2-479c-b6cb-02097a94b089\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6"
Jul 17 13:09:21.325316 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.325276    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6"
Jul 17 13:09:21.458057 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.458030    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6"]
Jul 17 13:09:21.460262 ip-10-0-137-99 kubenswrapper[2564]: W0717 13:09:21.460232    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa9678f_5bc2_479c_b6cb_02097a94b089.slice/crio-aefea142f6b5d4e11eaae7853cfd0ec85d2b74579406ceea1eb1a334f30a579a WatchSource:0}: Error finding container aefea142f6b5d4e11eaae7853cfd0ec85d2b74579406ceea1eb1a334f30a579a: Status 404 returned error can't find the container with id aefea142f6b5d4e11eaae7853cfd0ec85d2b74579406ceea1eb1a334f30a579a
Jul 17 13:09:21.532008 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.531966    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6" event={"ID":"faa9678f-5bc2-479c-b6cb-02097a94b089","Type":"ContainerStarted","Data":"45536db789fe7bcb810b5cabb15fc415e162862c71edadc301dd8e577286e924"}
Jul 17 13:09:21.532170 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:21.532050    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6" event={"ID":"faa9678f-5bc2-479c-b6cb-02097a94b089","Type":"ContainerStarted","Data":"aefea142f6b5d4e11eaae7853cfd0ec85d2b74579406ceea1eb1a334f30a579a"}
Jul 17 13:09:30.569958 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:30.569923    2564 generic.go:356] "Generic (PLEG): container finished" podID="faa9678f-5bc2-479c-b6cb-02097a94b089" containerID="45536db789fe7bcb810b5cabb15fc415e162862c71edadc301dd8e577286e924" exitCode=0
Jul 17 13:09:30.570379 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:30.569999    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6" event={"ID":"faa9678f-5bc2-479c-b6cb-02097a94b089","Type":"ContainerDied","Data":"45536db789fe7bcb810b5cabb15fc415e162862c71edadc301dd8e577286e924"}
Jul 17 13:09:31.576011 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:31.575971    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6" event={"ID":"faa9678f-5bc2-479c-b6cb-02097a94b089","Type":"ContainerStarted","Data":"ad9c9cc039a87924fc4c438432c91ef33c34d11cecc37d9276337d397e7fd27b"}
Jul 17 13:09:31.576420 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:31.576042    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6" event={"ID":"faa9678f-5bc2-479c-b6cb-02097a94b089","Type":"ContainerStarted","Data":"7553dc2b162c09315fa1dcdf005f8eb84ec2241190c6b44488eda08a4e2857c3"}
Jul 17 13:09:31.576420 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:31.576240    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6"
Jul 17 13:09:31.603454 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:31.603395    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6" podStartSLOduration=2.9429519060000002 podStartE2EDuration="11.603375033s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.660423127s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:09:31.600658378 +0000 UTC m=+2381.180770395" watchObservedRunningTime="2026-07-17 13:09:31.603375033 +0000 UTC m=+2381.183487056"
Jul 17 13:09:32.580122 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:32.580089    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6"
Jul 17 13:09:32.581359 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:32.581328    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6" podUID="faa9678f-5bc2-479c-b6cb-02097a94b089" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 17 13:09:33.584887 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:33.584838    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6" podUID="faa9678f-5bc2-479c-b6cb-02097a94b089" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 17 13:09:38.590048 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:38.590014    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6"
Jul 17 13:09:38.590570 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:38.590521    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6" podUID="faa9678f-5bc2-479c-b6cb-02097a94b089" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 17 13:09:48.591824 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:48.591791    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6"
Jul 17 13:09:50.964709 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:50.964664    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f958_7ab622df-86d3-4cc3-aba7-123f4bc0c996/ovn-acl-logging/0.log"
Jul 17 13:09:50.966348 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:50.966324    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f958_7ab622df-86d3-4cc3-aba7-123f4bc0c996/ovn-acl-logging/0.log"
Jul 17 13:09:51.069468 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:51.069429    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6"]
Jul 17 13:09:51.069769 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:51.069742    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6" podUID="faa9678f-5bc2-479c-b6cb-02097a94b089" containerName="kserve-container" containerID="cri-o://7553dc2b162c09315fa1dcdf005f8eb84ec2241190c6b44488eda08a4e2857c3" gracePeriod=30
Jul 17 13:09:51.069878 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:51.069780    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6" podUID="faa9678f-5bc2-479c-b6cb-02097a94b089" containerName="kube-rbac-proxy" containerID="cri-o://ad9c9cc039a87924fc4c438432c91ef33c34d11cecc37d9276337d397e7fd27b" gracePeriod=30
Jul 17 13:09:51.095473 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:51.095446    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:09:51.666396 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:51.666358    2564 generic.go:356] "Generic (PLEG): container finished" podID="faa9678f-5bc2-479c-b6cb-02097a94b089" containerID="ad9c9cc039a87924fc4c438432c91ef33c34d11cecc37d9276337d397e7fd27b" exitCode=2
Jul 17 13:09:51.666571 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:51.666425    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6" event={"ID":"faa9678f-5bc2-479c-b6cb-02097a94b089","Type":"ContainerDied","Data":"ad9c9cc039a87924fc4c438432c91ef33c34d11cecc37d9276337d397e7fd27b"}
Jul 17 13:09:52.013593 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.013567    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6"
Jul 17 13:09:52.163687 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.163653    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/faa9678f-5bc2-479c-b6cb-02097a94b089-kube-rbac-proxy-sar-config\") pod \"faa9678f-5bc2-479c-b6cb-02097a94b089\" (UID: \"faa9678f-5bc2-479c-b6cb-02097a94b089\") "
Jul 17 13:09:52.163919 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.163726    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/faa9678f-5bc2-479c-b6cb-02097a94b089-kserve-provision-location\") pod \"faa9678f-5bc2-479c-b6cb-02097a94b089\" (UID: \"faa9678f-5bc2-479c-b6cb-02097a94b089\") "
Jul 17 13:09:52.163919 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.163760    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/faa9678f-5bc2-479c-b6cb-02097a94b089-cabundle-cert\") pod \"faa9678f-5bc2-479c-b6cb-02097a94b089\" (UID: \"faa9678f-5bc2-479c-b6cb-02097a94b089\") "
Jul 17 13:09:52.163919 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.163803    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/faa9678f-5bc2-479c-b6cb-02097a94b089-proxy-tls\") pod \"faa9678f-5bc2-479c-b6cb-02097a94b089\" (UID: \"faa9678f-5bc2-479c-b6cb-02097a94b089\") "
Jul 17 13:09:52.163919 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.163874    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56wnj\" (UniqueName: \"kubernetes.io/projected/faa9678f-5bc2-479c-b6cb-02097a94b089-kube-api-access-56wnj\") pod \"faa9678f-5bc2-479c-b6cb-02097a94b089\" (UID: \"faa9678f-5bc2-479c-b6cb-02097a94b089\") "
Jul 17 13:09:52.164164 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.164143    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa9678f-5bc2-479c-b6cb-02097a94b089-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "faa9678f-5bc2-479c-b6cb-02097a94b089" (UID: "faa9678f-5bc2-479c-b6cb-02097a94b089"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:09:52.164238 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.164140    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa9678f-5bc2-479c-b6cb-02097a94b089-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "faa9678f-5bc2-479c-b6cb-02097a94b089" (UID: "faa9678f-5bc2-479c-b6cb-02097a94b089"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:09:52.164727 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.164680    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa9678f-5bc2-479c-b6cb-02097a94b089-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "faa9678f-5bc2-479c-b6cb-02097a94b089" (UID: "faa9678f-5bc2-479c-b6cb-02097a94b089"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 13:09:52.165930 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.165905    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa9678f-5bc2-479c-b6cb-02097a94b089-kube-api-access-56wnj" (OuterVolumeSpecName: "kube-api-access-56wnj") pod "faa9678f-5bc2-479c-b6cb-02097a94b089" (UID: "faa9678f-5bc2-479c-b6cb-02097a94b089"). InnerVolumeSpecName "kube-api-access-56wnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:09:52.166034 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.166014    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faa9678f-5bc2-479c-b6cb-02097a94b089-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "faa9678f-5bc2-479c-b6cb-02097a94b089" (UID: "faa9678f-5bc2-479c-b6cb-02097a94b089"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:09:52.265311 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.265207    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/faa9678f-5bc2-479c-b6cb-02097a94b089-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:09:52.265311 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.265246    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/faa9678f-5bc2-479c-b6cb-02097a94b089-kserve-provision-location\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:09:52.265311 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.265255    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/faa9678f-5bc2-479c-b6cb-02097a94b089-cabundle-cert\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:09:52.265311 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.265264    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/faa9678f-5bc2-479c-b6cb-02097a94b089-proxy-tls\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:09:52.265311 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.265272    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56wnj\" (UniqueName: \"kubernetes.io/projected/faa9678f-5bc2-479c-b6cb-02097a94b089-kube-api-access-56wnj\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:09:52.672833 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.672801    2564 generic.go:356] "Generic (PLEG): container finished" podID="faa9678f-5bc2-479c-b6cb-02097a94b089" containerID="7553dc2b162c09315fa1dcdf005f8eb84ec2241190c6b44488eda08a4e2857c3" exitCode=0
Jul 17 13:09:52.673002 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.672851    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6" event={"ID":"faa9678f-5bc2-479c-b6cb-02097a94b089","Type":"ContainerDied","Data":"7553dc2b162c09315fa1dcdf005f8eb84ec2241190c6b44488eda08a4e2857c3"}
Jul 17 13:09:52.673002 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.672856    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6"
Jul 17 13:09:52.673002 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.672907    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6" event={"ID":"faa9678f-5bc2-479c-b6cb-02097a94b089","Type":"ContainerDied","Data":"aefea142f6b5d4e11eaae7853cfd0ec85d2b74579406ceea1eb1a334f30a579a"}
Jul 17 13:09:52.673002 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.672934    2564 scope.go:117] "RemoveContainer" containerID="ad9c9cc039a87924fc4c438432c91ef33c34d11cecc37d9276337d397e7fd27b"
Jul 17 13:09:52.681842 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.681818    2564 scope.go:117] "RemoveContainer" containerID="7553dc2b162c09315fa1dcdf005f8eb84ec2241190c6b44488eda08a4e2857c3"
Jul 17 13:09:52.689597 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.689578    2564 scope.go:117] "RemoveContainer" containerID="45536db789fe7bcb810b5cabb15fc415e162862c71edadc301dd8e577286e924"
Jul 17 13:09:52.695628 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.695603    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6"]
Jul 17 13:09:52.696885 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.696865    2564 scope.go:117] "RemoveContainer" containerID="ad9c9cc039a87924fc4c438432c91ef33c34d11cecc37d9276337d397e7fd27b"
Jul 17 13:09:52.697153 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:09:52.697130    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad9c9cc039a87924fc4c438432c91ef33c34d11cecc37d9276337d397e7fd27b\": container with ID starting with ad9c9cc039a87924fc4c438432c91ef33c34d11cecc37d9276337d397e7fd27b not found: ID does not exist" containerID="ad9c9cc039a87924fc4c438432c91ef33c34d11cecc37d9276337d397e7fd27b"
Jul 17 13:09:52.697202 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.697160    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad9c9cc039a87924fc4c438432c91ef33c34d11cecc37d9276337d397e7fd27b"} err="failed to get container status \"ad9c9cc039a87924fc4c438432c91ef33c34d11cecc37d9276337d397e7fd27b\": rpc error: code = NotFound desc = could not find container \"ad9c9cc039a87924fc4c438432c91ef33c34d11cecc37d9276337d397e7fd27b\": container with ID starting with ad9c9cc039a87924fc4c438432c91ef33c34d11cecc37d9276337d397e7fd27b not found: ID does not exist"
Jul 17 13:09:52.697202 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.697180    2564 scope.go:117] "RemoveContainer" containerID="7553dc2b162c09315fa1dcdf005f8eb84ec2241190c6b44488eda08a4e2857c3"
Jul 17 13:09:52.697429 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:09:52.697412    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7553dc2b162c09315fa1dcdf005f8eb84ec2241190c6b44488eda08a4e2857c3\": container with ID starting with 7553dc2b162c09315fa1dcdf005f8eb84ec2241190c6b44488eda08a4e2857c3 not found: ID does not exist" containerID="7553dc2b162c09315fa1dcdf005f8eb84ec2241190c6b44488eda08a4e2857c3"
Jul 17 13:09:52.697499 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.697438    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7553dc2b162c09315fa1dcdf005f8eb84ec2241190c6b44488eda08a4e2857c3"} err="failed to get container status \"7553dc2b162c09315fa1dcdf005f8eb84ec2241190c6b44488eda08a4e2857c3\": rpc error: code = NotFound desc = could not find container \"7553dc2b162c09315fa1dcdf005f8eb84ec2241190c6b44488eda08a4e2857c3\": container with ID starting with 7553dc2b162c09315fa1dcdf005f8eb84ec2241190c6b44488eda08a4e2857c3 not found: ID does not exist"
Jul 17 13:09:52.697499 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.697459    2564 scope.go:117] "RemoveContainer" containerID="45536db789fe7bcb810b5cabb15fc415e162862c71edadc301dd8e577286e924"
Jul 17 13:09:52.697772 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:09:52.697755    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45536db789fe7bcb810b5cabb15fc415e162862c71edadc301dd8e577286e924\": container with ID starting with 45536db789fe7bcb810b5cabb15fc415e162862c71edadc301dd8e577286e924 not found: ID does not exist" containerID="45536db789fe7bcb810b5cabb15fc415e162862c71edadc301dd8e577286e924"
Jul 17 13:09:52.697819 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.697777    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45536db789fe7bcb810b5cabb15fc415e162862c71edadc301dd8e577286e924"} err="failed to get container status \"45536db789fe7bcb810b5cabb15fc415e162862c71edadc301dd8e577286e924\": rpc error: code = NotFound desc = could not find container \"45536db789fe7bcb810b5cabb15fc415e162862c71edadc301dd8e577286e924\": container with ID starting with 45536db789fe7bcb810b5cabb15fc415e162862c71edadc301dd8e577286e924 not found: ID does not exist"
Jul 17 13:09:52.701393 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:52.701370    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-jnpv6"]
Jul 17 13:09:53.014155 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:53.014075    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="faa9678f-5bc2-479c-b6cb-02097a94b089" path="/var/lib/kubelet/pods/faa9678f-5bc2-479c-b6cb-02097a94b089/volumes"
Jul 17 13:09:57.810528 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:57.810488    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-78gpt/must-gather-z8882"]
Jul 17 13:09:57.810957 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:57.810821    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faa9678f-5bc2-479c-b6cb-02097a94b089" containerName="kserve-container"
Jul 17 13:09:57.810957 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:57.810832    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="faa9678f-5bc2-479c-b6cb-02097a94b089" containerName="kserve-container"
Jul 17 13:09:57.810957 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:57.810852    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faa9678f-5bc2-479c-b6cb-02097a94b089" containerName="storage-initializer"
Jul 17 13:09:57.810957 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:57.810858    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="faa9678f-5bc2-479c-b6cb-02097a94b089" containerName="storage-initializer"
Jul 17 13:09:57.810957 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:57.810866    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faa9678f-5bc2-479c-b6cb-02097a94b089" containerName="kube-rbac-proxy"
Jul 17 13:09:57.810957 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:57.810872    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="faa9678f-5bc2-479c-b6cb-02097a94b089" containerName="kube-rbac-proxy"
Jul 17 13:09:57.810957 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:57.810926    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="faa9678f-5bc2-479c-b6cb-02097a94b089" containerName="kserve-container"
Jul 17 13:09:57.810957 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:57.810932    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="faa9678f-5bc2-479c-b6cb-02097a94b089" containerName="kube-rbac-proxy"
Jul 17 13:09:57.815730 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:57.815688    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-78gpt/must-gather-z8882"
Jul 17 13:09:57.818769 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:57.818748    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-78gpt\"/\"kube-root-ca.crt\""
Jul 17 13:09:57.818923 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:57.818804    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-78gpt\"/\"default-dockercfg-szmq7\""
Jul 17 13:09:57.819923 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:57.819907    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-78gpt\"/\"openshift-service-ca.crt\""
Jul 17 13:09:57.825876 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:57.825855    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:09:57.880314 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:57.880283    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-78gpt/must-gather-z8882"]
Jul 17 13:09:57.910916 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:57.910869    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e89800e4-1035-433c-88ca-1b41d2148a92-must-gather-output\") pod \"must-gather-z8882\" (UID: \"e89800e4-1035-433c-88ca-1b41d2148a92\") " pod="openshift-must-gather-78gpt/must-gather-z8882"
Jul 17 13:09:57.910916 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:57.910921    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l97vr\" (UniqueName: \"kubernetes.io/projected/e89800e4-1035-433c-88ca-1b41d2148a92-kube-api-access-l97vr\") pod \"must-gather-z8882\" (UID: \"e89800e4-1035-433c-88ca-1b41d2148a92\") " pod="openshift-must-gather-78gpt/must-gather-z8882"
Jul 17 13:09:57.943935 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:57.943899    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:09:58.012012 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:58.011978    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e89800e4-1035-433c-88ca-1b41d2148a92-must-gather-output\") pod \"must-gather-z8882\" (UID: \"e89800e4-1035-433c-88ca-1b41d2148a92\") " pod="openshift-must-gather-78gpt/must-gather-z8882"
Jul 17 13:09:58.012012 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:58.012012    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l97vr\" (UniqueName: \"kubernetes.io/projected/e89800e4-1035-433c-88ca-1b41d2148a92-kube-api-access-l97vr\") pod \"must-gather-z8882\" (UID: \"e89800e4-1035-433c-88ca-1b41d2148a92\") " pod="openshift-must-gather-78gpt/must-gather-z8882"
Jul 17 13:09:58.012328 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:58.012309    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e89800e4-1035-433c-88ca-1b41d2148a92-must-gather-output\") pod \"must-gather-z8882\" (UID: \"e89800e4-1035-433c-88ca-1b41d2148a92\") " pod="openshift-must-gather-78gpt/must-gather-z8882"
Jul 17 13:09:58.021924 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:58.021897    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l97vr\" (UniqueName: \"kubernetes.io/projected/e89800e4-1035-433c-88ca-1b41d2148a92-kube-api-access-l97vr\") pod \"must-gather-z8882\" (UID: \"e89800e4-1035-433c-88ca-1b41d2148a92\") " pod="openshift-must-gather-78gpt/must-gather-z8882"
Jul 17 13:09:58.139175 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:58.139083    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-78gpt/must-gather-z8882"
Jul 17 13:09:58.269557 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:58.269528    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-78gpt/must-gather-z8882"]
Jul 17 13:09:58.271408 ip-10-0-137-99 kubenswrapper[2564]: W0717 13:09:58.271379    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode89800e4_1035_433c_88ca_1b41d2148a92.slice/crio-377b8888f38d3a8b90429c33bac9f395dd3f53b21e36ac8e20223ded3b5bcdd2 WatchSource:0}: Error finding container 377b8888f38d3a8b90429c33bac9f395dd3f53b21e36ac8e20223ded3b5bcdd2: Status 404 returned error can't find the container with id 377b8888f38d3a8b90429c33bac9f395dd3f53b21e36ac8e20223ded3b5bcdd2
Jul 17 13:09:58.273206 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:58.273188    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 13:09:58.700506 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:09:58.700470    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-78gpt/must-gather-z8882" event={"ID":"e89800e4-1035-433c-88ca-1b41d2148a92","Type":"ContainerStarted","Data":"377b8888f38d3a8b90429c33bac9f395dd3f53b21e36ac8e20223ded3b5bcdd2"}
Jul 17 13:10:05.737771 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:05.737728    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-78gpt/must-gather-z8882" event={"ID":"e89800e4-1035-433c-88ca-1b41d2148a92","Type":"ContainerStarted","Data":"ad1d33d4d942abf27dc8d23b35d70395c17cec166c625a34a6343955c49b78bd"}
Jul 17 13:10:05.737771 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:05.737776    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-78gpt/must-gather-z8882" event={"ID":"e89800e4-1035-433c-88ca-1b41d2148a92","Type":"ContainerStarted","Data":"bd81c8bac0d56d33186faf52a56fa100cb17c81f2d85f009e2b2f1d24fc2f690"}
Jul 17 13:10:05.754948 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:05.754880    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-78gpt/must-gather-z8882" podStartSLOduration=2.322717459 podStartE2EDuration="8.7548573s" totalImagesPullingTime="6.432139841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:09:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:10:05.753919655 +0000 UTC m=+2415.334031673" watchObservedRunningTime="2026-07-17 13:10:05.7548573 +0000 UTC m=+2415.334969318"
Jul 17 13:10:25.826012 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:25.825970    2564 generic.go:356] "Generic (PLEG): container finished" podID="e89800e4-1035-433c-88ca-1b41d2148a92" containerID="bd81c8bac0d56d33186faf52a56fa100cb17c81f2d85f009e2b2f1d24fc2f690" exitCode=0
Jul 17 13:10:25.826438 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:25.826041    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-78gpt/must-gather-z8882" event={"ID":"e89800e4-1035-433c-88ca-1b41d2148a92","Type":"ContainerDied","Data":"bd81c8bac0d56d33186faf52a56fa100cb17c81f2d85f009e2b2f1d24fc2f690"}
Jul 17 13:10:25.826438 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:25.826329    2564 scope.go:117] "RemoveContainer" containerID="bd81c8bac0d56d33186faf52a56fa100cb17c81f2d85f009e2b2f1d24fc2f690"
Jul 17 13:10:26.328476 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:26.328450    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-78gpt_must-gather-z8882_e89800e4-1035-433c-88ca-1b41d2148a92/gather/0.log"
Jul 17 13:10:26.708389 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:26.708312    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:10:26.853959 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:26.853925    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:10:26.911440 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:26.911409    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:10:29.482107 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:29.482078    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9b4b4_c803ba62-1a8b-4b71-a355-fc4427be817b/global-pull-secret-syncer/0.log"
Jul 17 13:10:29.646149 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:29.646123    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log"
Jul 17 13:10:29.751934 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:29.751861    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-99.ec2.internal_f283087a6d8d1084ff25d034c2bbc6d1/haproxy/0.log"
Jul 17 13:10:31.686648 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:31.686614    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-78gpt/must-gather-z8882"]
Jul 17 13:10:31.687049 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:31.686831    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-78gpt/must-gather-z8882" podUID="e89800e4-1035-433c-88ca-1b41d2148a92" containerName="copy" containerID="cri-o://ad1d33d4d942abf27dc8d23b35d70395c17cec166c625a34a6343955c49b78bd" gracePeriod=2
Jul 17 13:10:31.691050 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:10:31.691015    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-z8882\" is forbidden: User \"system:node:ip-10-0-137-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-78gpt\": no relationship found between node 'ip-10-0-137-99.ec2.internal' and this object" podUID="e89800e4-1035-433c-88ca-1b41d2148a92" pod="openshift-must-gather-78gpt/must-gather-z8882"
Jul 17 13:10:31.692560 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:31.692535    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-78gpt/must-gather-z8882"]
Jul 17 13:10:31.852200 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:31.852174    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-78gpt_must-gather-z8882_e89800e4-1035-433c-88ca-1b41d2148a92/copy/0.log"
Jul 17 13:10:31.852593 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:31.852570    2564 generic.go:356] "Generic (PLEG): container finished" podID="e89800e4-1035-433c-88ca-1b41d2148a92" containerID="ad1d33d4d942abf27dc8d23b35d70395c17cec166c625a34a6343955c49b78bd" exitCode=143
Jul 17 13:10:31.927006 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:31.926985    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-78gpt_must-gather-z8882_e89800e4-1035-433c-88ca-1b41d2148a92/copy/0.log"
Jul 17 13:10:31.929648 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:31.929630    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-78gpt/must-gather-z8882"
Jul 17 13:10:31.932172 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:10:31.932148    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-z8882\" is forbidden: User \"system:node:ip-10-0-137-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-78gpt\": no relationship found between node 'ip-10-0-137-99.ec2.internal' and this object" podUID="e89800e4-1035-433c-88ca-1b41d2148a92" pod="openshift-must-gather-78gpt/must-gather-z8882"
Jul 17 13:10:32.004288 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:32.004213    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l97vr\" (UniqueName: \"kubernetes.io/projected/e89800e4-1035-433c-88ca-1b41d2148a92-kube-api-access-l97vr\") pod \"e89800e4-1035-433c-88ca-1b41d2148a92\" (UID: \"e89800e4-1035-433c-88ca-1b41d2148a92\") "
Jul 17 13:10:32.004288 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:32.004269    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e89800e4-1035-433c-88ca-1b41d2148a92-must-gather-output\") pod \"e89800e4-1035-433c-88ca-1b41d2148a92\" (UID: \"e89800e4-1035-433c-88ca-1b41d2148a92\") "
Jul 17 13:10:32.005690 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:32.005667    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89800e4-1035-433c-88ca-1b41d2148a92-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "e89800e4-1035-433c-88ca-1b41d2148a92" (UID: "e89800e4-1035-433c-88ca-1b41d2148a92"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 13:10:32.006449 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:32.006421    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e89800e4-1035-433c-88ca-1b41d2148a92-kube-api-access-l97vr" (OuterVolumeSpecName: "kube-api-access-l97vr") pod "e89800e4-1035-433c-88ca-1b41d2148a92" (UID: "e89800e4-1035-433c-88ca-1b41d2148a92"). InnerVolumeSpecName "kube-api-access-l97vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:10:32.104873 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:32.104837    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l97vr\" (UniqueName: \"kubernetes.io/projected/e89800e4-1035-433c-88ca-1b41d2148a92-kube-api-access-l97vr\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:10:32.104873 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:32.104868    2564 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e89800e4-1035-433c-88ca-1b41d2148a92-must-gather-output\") on node \"ip-10-0-137-99.ec2.internal\" DevicePath \"\""
Jul 17 13:10:32.446583 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:32.446549    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:10:32.553473 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:32.553446    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v7sc_146d6cf8-52ca-4482-9abb-ef40e770eb13/konnectivity-agent/0.log" limit=1024
Jul 17 13:10:32.858461 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:32.858435    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-78gpt_must-gather-z8882_e89800e4-1035-433c-88ca-1b41d2148a92/copy/0.log"
Jul 17 13:10:32.858915 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:32.858899    2564 scope.go:117] "RemoveContainer" containerID="ad1d33d4d942abf27dc8d23b35d70395c17cec166c625a34a6343955c49b78bd"
Jul 17 13:10:32.858990 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:32.858905    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-78gpt/must-gather-z8882"
Jul 17 13:10:32.861842 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:10:32.861803    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-z8882\" is forbidden: User \"system:node:ip-10-0-137-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-78gpt\": no relationship found between node 'ip-10-0-137-99.ec2.internal' and this object" podUID="e89800e4-1035-433c-88ca-1b41d2148a92" pod="openshift-must-gather-78gpt/must-gather-z8882"
Jul 17 13:10:32.867492 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:32.867472    2564 scope.go:117] "RemoveContainer" containerID="bd81c8bac0d56d33186faf52a56fa100cb17c81f2d85f009e2b2f1d24fc2f690"
Jul 17 13:10:32.869464 ip-10-0-137-99 kubenswrapper[2564]: E0717 13:10:32.869437    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-z8882\" is forbidden: User \"system:node:ip-10-0-137-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-78gpt\": no relationship found between node 'ip-10-0-137-99.ec2.internal' and this object" podUID="e89800e4-1035-433c-88ca-1b41d2148a92" pod="openshift-must-gather-78gpt/must-gather-z8882"
Jul 17 13:10:33.014465 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:33.014428    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e89800e4-1035-433c-88ca-1b41d2148a92" path="/var/lib/kubelet/pods/e89800e4-1035-433c-88ca-1b41d2148a92/volumes"
Jul 17 13:10:33.959891 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:33.959863    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-9dvjl_45cbefc2-d0c1-41ac-a78e-be14100134c4/kube-state-metrics/0.log"
Jul 17 13:10:33.986286 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:33.986199    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-9dvjl_45cbefc2-d0c1-41ac-a78e-be14100134c4/kube-rbac-proxy-main/0.log"
Jul 17 13:10:34.009717 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:34.009675    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-9dvjl_45cbefc2-d0c1-41ac-a78e-be14100134c4/kube-rbac-proxy-self/0.log"
Jul 17 13:10:34.107756 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:34.107723    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6jrn7_b9cd22df-55bb-4bfb-8108-8bbad86b57cd/node-exporter/0.log"
Jul 17 13:10:34.132692 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:34.132660    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6jrn7_b9cd22df-55bb-4bfb-8108-8bbad86b57cd/kube-rbac-proxy/0.log"
Jul 17 13:10:34.155882 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:34.155852    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6jrn7_b9cd22df-55bb-4bfb-8108-8bbad86b57cd/init-textfile/0.log"
Jul 17 13:10:34.459791 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:34.459762    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a147781b-2b14-4e9c-a69c-46b58bed2ee8/prometheus/0.log"
Jul 17 13:10:34.482742 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:34.482691    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a147781b-2b14-4e9c-a69c-46b58bed2ee8/config-reloader/0.log"
Jul 17 13:10:34.510884 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:34.510846    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a147781b-2b14-4e9c-a69c-46b58bed2ee8/thanos-sidecar/0.log"
Jul 17 13:10:34.535588 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:34.535565    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a147781b-2b14-4e9c-a69c-46b58bed2ee8/kube-rbac-proxy-web/0.log"
Jul 17 13:10:34.562969 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:34.562942    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a147781b-2b14-4e9c-a69c-46b58bed2ee8/kube-rbac-proxy/0.log"
Jul 17 13:10:34.588980 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:34.588958    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a147781b-2b14-4e9c-a69c-46b58bed2ee8/kube-rbac-proxy-thanos/0.log"
Jul 17 13:10:34.615355 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:34.615328    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a147781b-2b14-4e9c-a69c-46b58bed2ee8/init-config-reloader/0.log"
Jul 17 13:10:34.733423 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:34.733343    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-57bf6df894-8xmnn_e457f9d0-1192-4bfe-a11d-5bad8bc2c138/telemeter-client/0.log"
Jul 17 13:10:34.756695 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:34.756669    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-57bf6df894-8xmnn_e457f9d0-1192-4bfe-a11d-5bad8bc2c138/reload/0.log"
Jul 17 13:10:34.777454 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:34.777428    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-57bf6df894-8xmnn_e457f9d0-1192-4bfe-a11d-5bad8bc2c138/kube-rbac-proxy/0.log"
Jul 17 13:10:37.577752 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:37.577719    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-d8hr6/perf-node-gather-daemonset-wk9vp"]
Jul 17 13:10:37.578132 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:37.578019    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e89800e4-1035-433c-88ca-1b41d2148a92" containerName="gather"
Jul 17 13:10:37.578132 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:37.578030    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e89800e4-1035-433c-88ca-1b41d2148a92" containerName="gather"
Jul 17 13:10:37.578132 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:37.578041    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e89800e4-1035-433c-88ca-1b41d2148a92" containerName="copy"
Jul 17 13:10:37.578132 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:37.578047    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e89800e4-1035-433c-88ca-1b41d2148a92" containerName="copy"
Jul 17 13:10:37.578132 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:37.578103    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e89800e4-1035-433c-88ca-1b41d2148a92" containerName="gather"
Jul 17 13:10:37.578132 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:37.578109    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e89800e4-1035-433c-88ca-1b41d2148a92" containerName="copy"
Jul 17 13:10:37.583240 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:37.583217    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d8hr6/perf-node-gather-daemonset-wk9vp"
Jul 17 13:10:37.585990 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:37.585966    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-d8hr6\"/\"default-dockercfg-rxcjv\""
Jul 17 13:10:37.586132 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:37.585991    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d8hr6\"/\"openshift-service-ca.crt\""
Jul 17 13:10:37.586973 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:37.586958    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d8hr6\"/\"kube-root-ca.crt\""
Jul 17 13:10:37.590581 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:37.590559    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d8hr6/perf-node-gather-daemonset-wk9vp"]
Jul 17 13:10:37.750862 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:37.750818    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/16481f12-6d3e-437a-b92e-245d324961da-proc\") pod \"perf-node-gather-daemonset-wk9vp\" (UID: \"16481f12-6d3e-437a-b92e-245d324961da\") " pod="openshift-must-gather-d8hr6/perf-node-gather-daemonset-wk9vp"
Jul 17 13:10:37.751048 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:37.750886    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/16481f12-6d3e-437a-b92e-245d324961da-podres\") pod \"perf-node-gather-daemonset-wk9vp\" (UID: \"16481f12-6d3e-437a-b92e-245d324961da\") " pod="openshift-must-gather-d8hr6/perf-node-gather-daemonset-wk9vp"
Jul 17 13:10:37.751048 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:37.750912    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhrwj\" (UniqueName: \"kubernetes.io/projected/16481f12-6d3e-437a-b92e-245d324961da-kube-api-access-bhrwj\") pod \"perf-node-gather-daemonset-wk9vp\" (UID: \"16481f12-6d3e-437a-b92e-245d324961da\") " pod="openshift-must-gather-d8hr6/perf-node-gather-daemonset-wk9vp"
Jul 17 13:10:37.751048 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:37.750962    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16481f12-6d3e-437a-b92e-245d324961da-lib-modules\") pod \"perf-node-gather-daemonset-wk9vp\" (UID: \"16481f12-6d3e-437a-b92e-245d324961da\") " pod="openshift-must-gather-d8hr6/perf-node-gather-daemonset-wk9vp"
Jul 17 13:10:37.751048 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:37.750984    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16481f12-6d3e-437a-b92e-245d324961da-sys\") pod \"perf-node-gather-daemonset-wk9vp\" (UID: \"16481f12-6d3e-437a-b92e-245d324961da\") " pod="openshift-must-gather-d8hr6/perf-node-gather-daemonset-wk9vp"
Jul 17 13:10:37.851853 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:37.851767    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhrwj\" (UniqueName: \"kubernetes.io/projected/16481f12-6d3e-437a-b92e-245d324961da-kube-api-access-bhrwj\") pod \"perf-node-gather-daemonset-wk9vp\" (UID: \"16481f12-6d3e-437a-b92e-245d324961da\") " pod="openshift-must-gather-d8hr6/perf-node-gather-daemonset-wk9vp"
Jul 17 13:10:37.851853 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:37.851835    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16481f12-6d3e-437a-b92e-245d324961da-lib-modules\") pod \"perf-node-gather-daemonset-wk9vp\" (UID: \"16481f12-6d3e-437a-b92e-245d324961da\") " pod="openshift-must-gather-d8hr6/perf-node-gather-daemonset-wk9vp"
Jul 17 13:10:37.852037 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:37.851863    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16481f12-6d3e-437a-b92e-245d324961da-sys\") pod \"perf-node-gather-daemonset-wk9vp\" (UID: \"16481f12-6d3e-437a-b92e-245d324961da\") " pod="openshift-must-gather-d8hr6/perf-node-gather-daemonset-wk9vp"
Jul 17 13:10:37.852037 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:37.851898    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/16481f12-6d3e-437a-b92e-245d324961da-proc\") pod \"perf-node-gather-daemonset-wk9vp\" (UID: \"16481f12-6d3e-437a-b92e-245d324961da\") " pod="openshift-must-gather-d8hr6/perf-node-gather-daemonset-wk9vp"
Jul 17 13:10:37.852037 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:37.851956    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/16481f12-6d3e-437a-b92e-245d324961da-podres\") pod \"perf-node-gather-daemonset-wk9vp\" (UID: \"16481f12-6d3e-437a-b92e-245d324961da\") " pod="openshift-must-gather-d8hr6/perf-node-gather-daemonset-wk9vp"
Jul 17 13:10:37.852037 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:37.851991    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16481f12-6d3e-437a-b92e-245d324961da-sys\") pod \"perf-node-gather-daemonset-wk9vp\" (UID: \"16481f12-6d3e-437a-b92e-245d324961da\") " pod="openshift-must-gather-d8hr6/perf-node-gather-daemonset-wk9vp"
Jul 17 13:10:37.852037 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:37.852017    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/16481f12-6d3e-437a-b92e-245d324961da-proc\") pod \"perf-node-gather-daemonset-wk9vp\" (UID: \"16481f12-6d3e-437a-b92e-245d324961da\") " pod="openshift-must-gather-d8hr6/perf-node-gather-daemonset-wk9vp"
Jul 17 13:10:37.852195 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:37.852040    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16481f12-6d3e-437a-b92e-245d324961da-lib-modules\") pod \"perf-node-gather-daemonset-wk9vp\" (UID: \"16481f12-6d3e-437a-b92e-245d324961da\") " pod="openshift-must-gather-d8hr6/perf-node-gather-daemonset-wk9vp"
Jul 17 13:10:37.852195 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:37.852044    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/16481f12-6d3e-437a-b92e-245d324961da-podres\") pod \"perf-node-gather-daemonset-wk9vp\" (UID: \"16481f12-6d3e-437a-b92e-245d324961da\") " pod="openshift-must-gather-d8hr6/perf-node-gather-daemonset-wk9vp"
Jul 17 13:10:37.860856 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:37.860834    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhrwj\" (UniqueName: \"kubernetes.io/projected/16481f12-6d3e-437a-b92e-245d324961da-kube-api-access-bhrwj\") pod \"perf-node-gather-daemonset-wk9vp\" (UID: \"16481f12-6d3e-437a-b92e-245d324961da\") " pod="openshift-must-gather-d8hr6/perf-node-gather-daemonset-wk9vp"
Jul 17 13:10:37.894454 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:37.894420    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d8hr6/perf-node-gather-daemonset-wk9vp"
Jul 17 13:10:38.021320 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:38.021253    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d8hr6/perf-node-gather-daemonset-wk9vp"]
Jul 17 13:10:38.024303 ip-10-0-137-99 kubenswrapper[2564]: W0717 13:10:38.024273    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod16481f12_6d3e_437a_b92e_245d324961da.slice/crio-1914b56bd8344d56a524a2369262d6d5ba0965752b6b48171465c34aecfa9b4c WatchSource:0}: Error finding container 1914b56bd8344d56a524a2369262d6d5ba0965752b6b48171465c34aecfa9b4c: Status 404 returned error can't find the container with id 1914b56bd8344d56a524a2369262d6d5ba0965752b6b48171465c34aecfa9b4c
Jul 17 13:10:38.728014 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:38.727988    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wkj54_efb8a146-b5cf-4b54-bb64-3b060a736ca4/dns/0.log"
Jul 17 13:10:38.748980 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:38.748938    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wkj54_efb8a146-b5cf-4b54-bb64-3b060a736ca4/kube-rbac-proxy/0.log"
Jul 17 13:10:38.804435 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:38.804409    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g8rzg_688df482-0ac3-4a72-9632-c6a8c18b0b8d/dns-node-resolver/0.log"
Jul 17 13:10:38.885396 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:38.885362    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d8hr6/perf-node-gather-daemonset-wk9vp" event={"ID":"16481f12-6d3e-437a-b92e-245d324961da","Type":"ContainerStarted","Data":"1c36a531b6c226a0a400516d3271adf49c896a2eb20db008ee22343218248d80"}
Jul 17 13:10:38.885588 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:38.885415    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d8hr6/perf-node-gather-daemonset-wk9vp" event={"ID":"16481f12-6d3e-437a-b92e-245d324961da","Type":"ContainerStarted","Data":"1914b56bd8344d56a524a2369262d6d5ba0965752b6b48171465c34aecfa9b4c"}
Jul 17 13:10:38.885588 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:38.885445    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-d8hr6/perf-node-gather-daemonset-wk9vp"
Jul 17 13:10:38.902620 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:38.902566    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-d8hr6/perf-node-gather-daemonset-wk9vp" podStartSLOduration=1.902549899 podStartE2EDuration="1.902549899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 13:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:10:38.901247915 +0000 UTC m=+2448.481359934" watchObservedRunningTime="2026-07-17 13:10:38.902549899 +0000 UTC m=+2448.482661918"
Jul 17 13:10:39.328308 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:39.328274    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xxld5_cc19f981-8855-47ce-98c7-6e8e9c0e65d2/node-ca/0.log"
Jul 17 13:10:40.520613 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:40.520582    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4h4tb_ed27f8cc-6874-4594-905b-2483d9f9463d/serve-healthcheck-canary/0.log"
Jul 17 13:10:40.922020 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:40.921990    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-2pclj_6652240d-b5e1-4079-b293-3e5d8469b4f9/insights-operator/0.log"
Jul 17 13:10:40.922791 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:40.922774    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-2pclj_6652240d-b5e1-4079-b293-3e5d8469b4f9/insights-operator/1.log"
Jul 17 13:10:41.112236 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:41.112206    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vd7js_9fe27e83-0d04-4f44-9b37-f410e08e5f51/kube-rbac-proxy/0.log"
Jul 17 13:10:41.135761 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:41.135732    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vd7js_9fe27e83-0d04-4f44-9b37-f410e08e5f51/exporter/0.log"
Jul 17 13:10:41.159627 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:41.159603    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vd7js_9fe27e83-0d04-4f44-9b37-f410e08e5f51/extractor/0.log"
Jul 17 13:10:43.198266 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:43.198234    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-6697cdd9c7-mk8zm_d81d19e0-0bb8-4932-ac57-b8dfb63712df/manager/0.log"
Jul 17 13:10:43.276076 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:43.276050    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-kvtwn_580a2df3-d616-43ab-a3c4-a7ba1da50156/server/0.log"
Jul 17 13:10:43.375257 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:43.375222    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-hwtrx_5552b3ce-f442-4af3-a184-1b0d8e7466d4/s3-init/0.log"
Jul 17 13:10:43.392500 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:43.392473    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-hwtrx_5552b3ce-f442-4af3-a184-1b0d8e7466d4/download-hf-model/0.log"
Jul 17 13:10:43.422335 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:43.422300    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-6br26_c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae/s3-tls-init-custom/0.log"
Jul 17 13:10:43.438269 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:43.438238    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-6br26_c5c37dbb-0cf4-4ceb-87b8-a331a4a7eaae/download-hf-model/0.log"
Jul 17 13:10:43.466413 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:43.466320    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-wvhnf_a7aa9fd2-1266-444b-baec-91290b7ea5d7/s3-tls-init-serving/0.log"
Jul 17 13:10:43.482853 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:43.482826    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-wvhnf_a7aa9fd2-1266-444b-baec-91290b7ea5d7/download-hf-model/0.log"
Jul 17 13:10:43.508326 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:43.508304    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-xmj27_d233aa4a-fed8-47cb-afb6-3513512422ad/seaweedfs/0.log"
Jul 17 13:10:43.532146 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:43.532118    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-htg94_9cb42b44-51b0-4a13-8b7d-45069950b35f/seaweedfs-tls-custom/0.log"
Jul 17 13:10:43.556116 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:43.556088    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-9cj2q_3c6c3bed-ad1b-48b4-b004-cbc7258e88a2/seaweedfs-tls-serving/0.log"
Jul 17 13:10:44.898976 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:44.898947    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-d8hr6/perf-node-gather-daemonset-wk9vp"
Jul 17 13:10:47.522629 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:47.522599    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-xmzkp_b937bdf9-4b6d-4fd0-aa58-7e721cdd064e/migrator/0.log"
Jul 17 13:10:47.545415 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:47.545328    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-xmzkp_b937bdf9-4b6d-4fd0-aa58-7e721cdd064e/graceful-termination/0.log"
Jul 17 13:10:47.857659 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:47.857567    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-42w44_a6b7914b-3758-4e34-bc34-309714b615c6/kube-storage-version-migrator-operator/1.log"
Jul 17 13:10:47.858474 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:47.858453    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-42w44_a6b7914b-3758-4e34-bc34-309714b615c6/kube-storage-version-migrator-operator/0.log"
Jul 17 13:10:48.793244 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:48.793212    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7ql7p_c47b98e9-a381-4815-b8d9-4737cf123e0d/kube-multus/0.log"
Jul 17 13:10:49.016913 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:49.016884    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sld25_b09e01d5-4dca-4a57-903d-f6daacfdb5d9/kube-multus-additional-cni-plugins/0.log"
Jul 17 13:10:49.036980 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:49.036954    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sld25_b09e01d5-4dca-4a57-903d-f6daacfdb5d9/egress-router-binary-copy/0.log"
Jul 17 13:10:49.059624 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:49.059601    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sld25_b09e01d5-4dca-4a57-903d-f6daacfdb5d9/cni-plugins/0.log"
Jul 17 13:10:49.079904 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:49.079876    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sld25_b09e01d5-4dca-4a57-903d-f6daacfdb5d9/bond-cni-plugin/0.log"
Jul 17 13:10:49.100094 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:49.100069    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sld25_b09e01d5-4dca-4a57-903d-f6daacfdb5d9/routeoverride-cni/0.log"
Jul 17 13:10:49.121554 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:49.121531    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sld25_b09e01d5-4dca-4a57-903d-f6daacfdb5d9/whereabouts-cni-bincopy/0.log"
Jul 17 13:10:49.141150 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:49.141121    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sld25_b09e01d5-4dca-4a57-903d-f6daacfdb5d9/whereabouts-cni/0.log"
Jul 17 13:10:49.481515 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:49.481490    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lxrxw_937e7356-528f-4a56-a80e-9486657ec301/network-metrics-daemon/0.log"
Jul 17 13:10:49.525995 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:49.525967    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lxrxw_937e7356-528f-4a56-a80e-9486657ec301/kube-rbac-proxy/0.log"
Jul 17 13:10:50.242052 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:50.242023    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f958_7ab622df-86d3-4cc3-aba7-123f4bc0c996/ovn-controller/0.log"
Jul 17 13:10:50.262793 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:50.262758    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f958_7ab622df-86d3-4cc3-aba7-123f4bc0c996/ovn-acl-logging/1.log"
Jul 17 13:10:50.262985 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:50.262789    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f958_7ab622df-86d3-4cc3-aba7-123f4bc0c996/ovn-acl-logging/0.log"
Jul 17 13:10:50.284862 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:50.284836    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f958_7ab622df-86d3-4cc3-aba7-123f4bc0c996/kube-rbac-proxy-node/0.log"
Jul 17 13:10:50.305830 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:50.305805    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f958_7ab622df-86d3-4cc3-aba7-123f4bc0c996/kube-rbac-proxy-ovn-metrics/0.log"
Jul 17 13:10:50.326206 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:50.326179    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f958_7ab622df-86d3-4cc3-aba7-123f4bc0c996/northd/0.log"
Jul 17 13:10:50.347042 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:50.347022    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f958_7ab622df-86d3-4cc3-aba7-123f4bc0c996/nbdb/0.log"
Jul 17 13:10:50.372926 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:50.372892    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f958_7ab622df-86d3-4cc3-aba7-123f4bc0c996/sbdb/0.log"
Jul 17 13:10:50.415809 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:50.415772    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f958_7ab622df-86d3-4cc3-aba7-123f4bc0c996/ovnkube-controller/0.log"
Jul 17 13:10:51.876217 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:51.876185    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-xl2np_a72fcab3-1981-4637-a041-225335390f1d/network-check-target-container/0.log"
Jul 17 13:10:52.795317 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:52.795292    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-xdqkr_fe8942fd-34dd-4b5b-ba05-0c6488374820/iptables-alerter/0.log"
Jul 17 13:10:53.473582 ip-10-0-137-99 kubenswrapper[2564]: I0717 13:10:53.473502    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-vtsx2_f92fb403-4621-413f-9e6b-c041411f774a/tuned/0.log"