Sep 17 20:47:13.185118 ip-10-0-138-140 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 20:47:13.185129 ip-10-0-138-140 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 20:47:13.185136 ip-10-0-138-140 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 20:47:13.185281 ip-10-0-138-140 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 20:47:23.401314 ip-10-0-138-140 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 20:47:23.401328 ip-10-0-138-140 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot e55f306cbfc146588b002f97ec7963eb --
Sep 17 20:49:16.962509 ip-10-0-138-140 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 20:49:20.284574 ip-10-0-138-140 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 20:49:20.284574 ip-10-0-138-140 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 20:49:20.284574 ip-10-0-138-140 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 20:49:20.284574 ip-10-0-138-140 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 20:49:20.284574 ip-10-0-138-140 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 20:49:20.285303 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.284661    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 20:49:20.287999 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.287983    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:20.287999 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.287999    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:20.288061 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288003    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:20.288061 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288007    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:20.288061 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288010    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:20.288061 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288013    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:20.288061 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288016    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:20.288061 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288019    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:20.288061 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288022    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:20.288061 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288025    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:20.288061 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288028    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:20.288061 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288030    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:20.288061 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288033    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:20.288061 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288036    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:20.288061 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288039    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:20.288061 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288041    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:20.288061 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288049    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:20.288061 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288051    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:20.288061 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288054    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:20.288061 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288057    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:20.288061 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288062    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:20.288534 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288067    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:20.288534 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288070    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:20.288534 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288073    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:20.288534 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288077    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:20.288534 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288081    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:20.288534 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288085    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:20.288534 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288087    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:20.288534 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288091    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:20.288534 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288094    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:20.288534 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288096    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:20.288534 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288099    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:20.288534 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288102    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:20.288534 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288104    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:20.288534 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288107    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:20.288534 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288110    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:20.288534 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288112    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:20.288534 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288115    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:20.288534 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288117    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:20.288534 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288122    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:20.289004 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288125    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:20.289004 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288129    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:20.289004 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288132    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:20.289004 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288134    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:20.289004 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288137    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:20.289004 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288140    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:20.289004 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288143    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:20.289004 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288146    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:20.289004 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288149    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:20.289004 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288152    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:20.289004 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288154    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:20.289004 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288157    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:20.289004 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288160    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:20.289004 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288163    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:20.289004 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288166    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:20.289004 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288169    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:20.289004 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288172    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:20.289004 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288175    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:20.289004 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288178    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:20.289514 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288180    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:20.289514 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288183    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:20.289514 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288185    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:20.289514 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288188    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:20.289514 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288191    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:20.289514 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288193    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:20.289514 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288196    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:20.289514 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288199    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:20.289514 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288201    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:20.289514 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288204    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:20.289514 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288220    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:20.289514 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288223    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:20.289514 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288226    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:20.289514 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288229    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:20.289514 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288232    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:20.289514 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288235    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:20.289514 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288238    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:20.289514 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288240    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:20.289514 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288243    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:20.289514 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288245    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:20.290033 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288248    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:20.290033 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288251    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:20.290033 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288253    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:20.290033 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288256    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:20.290033 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288259    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:20.290033 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288263    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:20.290033 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288265    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:20.290033 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288269    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:20.290033 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288272    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:20.290033 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288274    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:20.290033 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288292    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:20.290033 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288296    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:20.290033 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288299    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:20.290033 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288302    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:20.290033 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288305    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:20.290033 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288308    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:20.290033 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288311    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:20.290033 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288314    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:20.290033 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288317    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:20.290033 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288319    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:20.290551 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288977    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:20.290551 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288985    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:20.290551 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288990    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:20.290551 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288993    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:20.290551 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.288996    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:20.290551 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289001    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:20.290551 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289005    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:20.290551 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289008    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:20.290551 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289011    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:20.290551 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289014    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:20.290551 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289018    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:20.290551 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289022    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:20.290551 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289025    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:20.290551 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289028    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:20.290551 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289031    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:20.290551 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289034    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:20.290551 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289037    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:20.290551 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289040    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:20.290995 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289042    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:20.290995 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289046    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:20.290995 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289048    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:20.290995 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289051    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:20.290995 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289054    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:20.290995 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289057    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:20.290995 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289059    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:20.290995 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289062    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:20.290995 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289065    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:20.290995 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289067    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:20.290995 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289070    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:20.290995 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289073    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:20.290995 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289075    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:20.290995 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289078    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:20.290995 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289081    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:20.290995 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289083    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:20.290995 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289086    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:20.290995 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289089    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:20.290995 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289091    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:20.290995 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289094    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:20.291498 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289097    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:20.291498 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289099    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:20.291498 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289102    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:20.291498 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289105    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:20.291498 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289107    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:20.291498 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289110    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:20.291498 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289113    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:20.291498 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289115    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:20.291498 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289118    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:20.291498 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289120    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:20.291498 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289124    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:20.291498 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289126    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:20.291498 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289129    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:20.291498 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289132    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:20.291498 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289135    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:20.291498 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289138    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:20.291498 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289141    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:20.291498 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289143    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:20.291498 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289146    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:20.291498 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289149    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:20.291979 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289152    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:20.291979 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289154    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:20.291979 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289158    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:20.291979 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289160    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:20.291979 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289163    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:20.291979 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289166    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:20.291979 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289168    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:20.291979 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289171    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:20.291979 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289174    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:20.291979 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289176    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:20.291979 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289179    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:20.291979 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289182    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:20.291979 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289184    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:20.291979 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289188    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:20.291979 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289191    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:20.291979 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289194    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:20.291979 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289196    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:20.291979 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289199    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:20.291979 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289201    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:20.291979 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289204    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:20.292479 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289221    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:20.292479 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289223    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:20.292479 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289227    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:20.292479 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289229    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:20.292479 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289233    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:20.292479 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289239    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:20.292479 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289241    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:20.292479 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289244    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:20.292479 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289247    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:20.292479 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289250    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:20.292479 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289253    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:20.292479 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289256    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:20.292479 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289258    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:20.292479 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289261    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:20.292479 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289264    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:20.292479 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289267    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:20.292479 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289269    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:20.292479 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289272    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:20.292479 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289275    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:20.292479 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289277    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:20.292955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289280    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:20.292955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289640    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:20.292955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289644    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:20.292955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289647    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:20.292955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289650    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:20.292955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289654    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:20.292955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289656    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:20.292955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289659    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:20.292955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289662    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:20.292955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289664    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:20.292955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289667    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:20.292955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289670    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:20.292955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289673    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:20.292955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289675    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:20.292955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289679    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:20.292955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289682    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:20.292955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289684    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:20.292955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289689    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:20.292955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289692    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:20.292955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289694    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:20.293463 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289697    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:20.293463 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289700    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:20.293463 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289703    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:20.293463 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289705    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:20.293463 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289708    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:20.293463 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289711    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:20.293463 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289714    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:20.293463 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289716    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:20.293463 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289719    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:20.293463 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289722    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:20.293463 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289725    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:20.293463 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289727    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:20.293463 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289730    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:20.293463 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289733    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:20.293463 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289736    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:20.293463 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289738    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:20.293463 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289741    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:20.293463 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289745    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:20.293463 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289750    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:20.293955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289753    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:20.293955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289756    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:20.293955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289759    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:20.293955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289762    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:20.293955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289764    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:20.293955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289767    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:20.293955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289770    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:20.293955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289773    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:20.293955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289776    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:20.293955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289779    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:20.293955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289783    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:20.293955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289786    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:20.293955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289788    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:20.293955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289791    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:20.293955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289793    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:20.293955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289796    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:20.293955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289800    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:20.293955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289805    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:20.293955 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289808    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:20.294440 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289811    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:20.294440 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289814    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:20.294440 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289816    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:20.294440 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289819    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:20.294440 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289822    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:20.294440 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289825    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:20.294440 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289827    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:20.294440 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289830    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:20.294440 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289833    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:20.294440 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289836    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:20.294440 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289838    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:20.294440 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289841    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:20.294440 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289845    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:20.294440 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289849    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:20.294440 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289851    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:20.294440 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289854    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:20.294440 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289857    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:20.294440 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289860    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:20.294440 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289863    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:20.294440 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289865    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:20.294923 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289868    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:20.294923 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289870    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:20.294923 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289873    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:20.294923 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289877    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:20.294923 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289879    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:20.294923 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289882    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:20.294923 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289885    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:20.294923 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289887    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:20.294923 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289890    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:20.294923 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289893    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:20.294923 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289895    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:20.294923 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289898    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:20.294923 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289902    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:20.294923 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289905    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:20.294923 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289908    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:20.294923 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289911    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:20.294923 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289914    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:20.294923 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289917    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:20.294923 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289920    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:20.294923 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289923    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:20.295422 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289925    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:20.295422 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.289928    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:20.295422 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290013    2566 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 20:49:20.295422 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290053    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 20:49:20.295422 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290062    2566 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 20:49:20.295422 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290067    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 20:49:20.295422 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290072    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 20:49:20.295422 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290076    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 20:49:20.295422 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290080    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 20:49:20.295422 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290085    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 20:49:20.295422 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290088    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 20:49:20.295422 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290091    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 20:49:20.295422 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290095    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 20:49:20.295422 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290099    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 20:49:20.295422 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290102    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 20:49:20.295422 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290106    2566 flags.go:64] FLAG: --cgroup-root=""
Sep 17 20:49:20.295422 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290109    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 20:49:20.295422 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290112    2566 flags.go:64] FLAG: --client-ca-file=""
Sep 17 20:49:20.295422 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290115    2566 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 20:49:20.295422 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290118    2566 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 20:49:20.295422 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290122    2566 flags.go:64] FLAG: --cluster-domain=""
Sep 17 20:49:20.295422 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290125    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 20:49:20.295422 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290128    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 20:49:20.295422 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290131    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 20:49:20.295992 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290134    2566 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 20:49:20.295992 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290138    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 20:49:20.295992 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290141    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 20:49:20.295992 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290144    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 20:49:20.295992 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290147    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 20:49:20.295992 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290150    2566 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 20:49:20.295992 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290153    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 20:49:20.295992 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290156    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 20:49:20.295992 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290159    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 20:49:20.295992 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290163    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 20:49:20.295992 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290167    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 20:49:20.295992 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290170    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 20:49:20.295992 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290174    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 20:49:20.295992 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290177    2566 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 20:49:20.295992 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290180    2566 flags.go:64] FLAG: --enable-server="true"
Sep 17 20:49:20.295992 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290183    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 20:49:20.295992 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290188    2566 flags.go:64] FLAG: --event-burst="100"
Sep 17 20:49:20.295992 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290191    2566 flags.go:64] FLAG: --event-qps="50"
Sep 17 20:49:20.295992 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290195    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 20:49:20.295992 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290198    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 20:49:20.295992 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290201    2566 flags.go:64] FLAG: --eviction-hard=""
Sep 17 20:49:20.295992 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290218    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 20:49:20.295992 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290221    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 20:49:20.295992 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290225    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 20:49:20.295992 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290228    2566 flags.go:64] FLAG: --eviction-soft=""
Sep 17 20:49:20.296600 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290231    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 20:49:20.296600 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290235    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 20:49:20.296600 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290238    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 20:49:20.296600 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290241    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 20:49:20.296600 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290244    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 20:49:20.296600 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290247    2566 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 20:49:20.296600 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290250    2566 flags.go:64] FLAG: --feature-gates=""
Sep 17 20:49:20.296600 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290253    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 20:49:20.296600 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290256    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 20:49:20.296600 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290260    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 20:49:20.296600 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290263    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 20:49:20.296600 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290266    2566 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 20:49:20.296600 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290269    2566 flags.go:64] FLAG: --help="false"
Sep 17 20:49:20.296600 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290272    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-138-140.ec2.internal"
Sep 17 20:49:20.296600 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290275    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 20:49:20.296600 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290278    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 20:49:20.296600 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290282    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 20:49:20.296600 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290285    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 20:49:20.296600 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290288    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 20:49:20.296600 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290293    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 20:49:20.296600 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290296    2566 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 20:49:20.296600 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290299    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 20:49:20.296600 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290302    2566 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 20:49:20.297175 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290306    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 20:49:20.297175 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290309    2566 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 20:49:20.297175 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290312    2566 flags.go:64] FLAG: --kube-reserved=""
Sep 17 20:49:20.297175 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290315    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 20:49:20.297175 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290325    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 20:49:20.297175 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290329    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 20:49:20.297175 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290332    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 20:49:20.297175 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290335    2566 flags.go:64] FLAG: --lock-file=""
Sep 17 20:49:20.297175 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290338    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 20:49:20.297175 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290341    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 20:49:20.297175 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290345    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 20:49:20.297175 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290350    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 20:49:20.297175 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290353    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 20:49:20.297175 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290356    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 20:49:20.297175 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290359    2566 flags.go:64] FLAG: --logging-format="text"
Sep 17 20:49:20.297175 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290362    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 20:49:20.297175 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290366    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 20:49:20.297175 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290368    2566 flags.go:64] FLAG: --manifest-url=""
Sep 17 20:49:20.297175 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290372    2566 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 20:49:20.297175 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290376    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 20:49:20.297175 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290379    2566 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 20:49:20.297175 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290384    2566 flags.go:64] FLAG: --max-pods="110"
Sep 17 20:49:20.297175 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290387    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 20:49:20.297175 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290390    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 20:49:20.297175 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290393    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 20:49:20.297832 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290397    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 20:49:20.297832 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290400    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 20:49:20.297832 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290403    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 20:49:20.297832 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290407    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 20:49:20.297832 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290415    2566 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 20:49:20.297832 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290419    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 20:49:20.297832 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290422    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 20:49:20.297832 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290426    2566 flags.go:64] FLAG: --pod-cidr=""
Sep 17 20:49:20.297832 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290429    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 20:49:20.297832 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290434    2566 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 20:49:20.297832 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290438    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 20:49:20.297832 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290441    2566 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 20:49:20.297832 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290444    2566 flags.go:64] FLAG: --port="10250"
Sep 17 20:49:20.297832 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290447    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 20:49:20.297832 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290450    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08ac43c9a7b390931"
Sep 17 20:49:20.297832 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290453    2566 flags.go:64] FLAG: --qos-reserved=""
Sep 17 20:49:20.297832 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290456    2566 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 20:49:20.297832 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290460    2566 flags.go:64] FLAG: --register-node="true"
Sep 17 20:49:20.297832 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290463    2566 flags.go:64] FLAG: --register-with-taints=""
Sep 17 20:49:20.297832 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290466    2566 flags.go:64] FLAG: --registry-burst="10"
Sep 17 20:49:20.297832 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290469    2566 flags.go:64] FLAG: --registry-qps="5"
Sep 17 20:49:20.297832 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290472    2566 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 20:49:20.297832 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290475    2566 flags.go:64] FLAG: --reserved-memory=""
Sep 17 20:49:20.297832 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290479    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 20:49:20.297832 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290482    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 20:49:20.298469 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290485    2566 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 20:49:20.298469 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290488    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 20:49:20.298469 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290491    2566 flags.go:64] FLAG: --runonce="false"
Sep 17 20:49:20.298469 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290494    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 20:49:20.298469 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290497    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 20:49:20.298469 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290500    2566 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 20:49:20.298469 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290503    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 20:49:20.298469 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290506    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 20:49:20.298469 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290509    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 20:49:20.298469 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290512    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 20:49:20.298469 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290517    2566 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 20:49:20.298469 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290521    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 20:49:20.298469 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290524    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 20:49:20.298469 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290526    2566 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 20:49:20.298469 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290531    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 20:49:20.298469 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290534    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 20:49:20.298469 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290538    2566 flags.go:64] FLAG: --system-cgroups=""
Sep 17 20:49:20.298469 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290542    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 20:49:20.298469 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290550    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 20:49:20.298469 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290555    2566 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 20:49:20.298469 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290559    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 20:49:20.298469 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290563    2566 flags.go:64] FLAG: --tls-min-version=""
Sep 17 20:49:20.298469 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290566    2566 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 20:49:20.298469 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290569    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 20:49:20.298469 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290572    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 20:49:20.299076 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290576    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 20:49:20.299076 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290579    2566 flags.go:64] FLAG: --v="2"
Sep 17 20:49:20.299076 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290583    2566 flags.go:64] FLAG: --version="false"
Sep 17 20:49:20.299076 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290587    2566 flags.go:64] FLAG: --vmodule=""
Sep 17 20:49:20.299076 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290591    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 20:49:20.299076 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.290594    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 20:49:20.299076 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290709    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:20.299076 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290713    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:20.299076 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290717    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:20.299076 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290720    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:20.299076 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290723    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:20.299076 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290726    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:20.299076 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290729    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:20.299076 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290732    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:20.299076 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290734    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:20.299076 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290737    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:20.299076 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290740    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:20.299076 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290743    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:20.299076 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290746    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:20.299076 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290749    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:20.299076 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290752    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:20.299076 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290756    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:20.299639 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290759    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:20.299639 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290762    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:20.299639 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290765    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:20.299639 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290768    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:20.299639 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290771    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:20.299639 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290773    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:20.299639 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290776    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:20.299639 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290780    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:20.299639 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290784    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:20.299639 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290787    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:20.299639 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290790    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:20.299639 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290793    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:20.299639 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290795    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:20.299639 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290798    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:20.299639 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290800    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:20.299639 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290803    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:20.299639 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290806    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:20.299639 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290809    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:20.299639 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290811    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:20.300115 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290814    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:20.300115 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290817    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:20.300115 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290819    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:20.300115 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290822    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:20.300115 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290824    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:20.300115 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290827    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:20.300115 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290830    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:20.300115 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290834    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:20.300115 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290838    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:20.300115 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290841    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:20.300115 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290844    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:20.300115 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290847    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:20.300115 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290851    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:20.300115 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290854    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:20.300115 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290857    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:20.300115 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290860    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:20.300115 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290863    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:20.300115 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290866    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:20.300115 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290869    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:20.300598 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290871    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:20.300598 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290874    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:20.300598 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290877    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:20.300598 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290879    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:20.300598 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290882    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:20.300598 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290885    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:20.300598 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290888    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:20.300598 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290890    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:20.300598 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290893    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:20.300598 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290896    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:20.300598 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290898    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:20.300598 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290901    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:20.300598 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290904    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:20.300598 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290907    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:20.300598 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290909    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:20.300598 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290912    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:20.300598 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290914    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:20.300598 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290917    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:20.300598 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290920    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:20.300598 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290922    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:20.301091 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290925    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:20.301091 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290929    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:20.301091 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290933    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:20.301091 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290936    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:20.301091 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290938    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:20.301091 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290943    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:20.301091 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290946    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:20.301091 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290949    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:20.301091 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290952    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:20.301091 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290954    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:20.301091 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290958    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:20.301091 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290960    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:20.301091 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290963    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:20.301091 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290966    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:20.301091 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290968    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:20.301091 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290971    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:20.301091 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290974    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:20.301091 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290976    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:20.301091 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290979    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:20.301565 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290982    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:20.301565 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290985    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:20.301565 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290987    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:20.301565 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290990    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:20.301565 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290992    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:20.301565 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.290995    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:20.301565 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.291465    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 20:49:20.301565 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.298132    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 20:49:20.301565 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.298147    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 20:49:20.301565 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.298168    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 20:49:20.301565 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.298174    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 20:49:20.301565 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298244    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:20.301565 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298250    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:20.301565 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298254    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:20.301565 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298257    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:20.301961 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298260    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:20.301961 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298263    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:20.301961 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298266    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:20.301961 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298269    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:20.301961 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298272    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:20.301961 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298274    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:20.301961 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298277    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:20.301961 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298280    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:20.301961 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298283    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:20.301961 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298286    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:20.301961 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298289    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:20.301961 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298292    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:20.301961 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298294    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:20.301961 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298297    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:20.301961 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298300    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:20.301961 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298303    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:20.301961 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298308    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:20.301961 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298312    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:20.301961 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298323    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:20.302448 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298327    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:20.302448 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298330    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:20.302448 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298333    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:20.302448 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298336    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:20.302448 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298339    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:20.302448 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298343    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:20.302448 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298347    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:20.302448 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298350    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:20.302448 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298352    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:20.302448 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298355    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:20.302448 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298358    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:20.302448 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298361    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:20.302448 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298364    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:20.302448 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298366    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:20.302448 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298369    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:20.302448 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298372    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:20.302448 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298375    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:20.302448 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298378    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:20.302448 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298380    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:20.302448 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298383    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:20.302949 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298386    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:20.302949 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298390    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:20.302949 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298394    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:20.302949 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298397    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:20.302949 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298400    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:20.302949 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298403    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:20.302949 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298405    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:20.302949 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298408    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:20.302949 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298411    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:20.302949 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298413    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:20.302949 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298416    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:20.302949 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298419    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:20.302949 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298423    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:20.302949 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298426    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:20.302949 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298429    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:20.302949 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298433    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:20.302949 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298435    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:20.302949 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298439    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:20.302949 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298442    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:20.303902 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298445    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:20.303902 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298448    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:20.303902 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298450    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:20.303902 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298453    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:20.303902 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298456    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:20.303902 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298458    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:20.303902 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298461    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:20.303902 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298464    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:20.303902 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298467    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:20.303902 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298469    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:20.303902 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298472    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:20.303902 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298475    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:20.303902 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298478    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:20.303902 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298481    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:20.303902 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298484    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:20.303902 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298486    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:20.303902 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298490    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:20.303902 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298492    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:20.303902 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298495    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:20.303902 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298498    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:20.304616 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298500    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:20.304616 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298503    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:20.304616 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298506    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:20.304616 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298508    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:20.304616 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298511    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:20.304616 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298514    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:20.304616 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298517    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:20.304616 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298519    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:20.304616 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298522    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:20.304616 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298525    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:20.304616 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298528    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:20.304616 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298531    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:20.304616 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298534    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:20.304616 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298537    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:20.304616 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298539    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:20.304616 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298542    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:20.304616 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298544    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:20.305038 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.298549    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 20:49:20.305038 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298666    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:20.305038 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298671    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:20.305038 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298674    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:20.305038 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298677    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:20.305038 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298680    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:20.305038 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298683    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:20.305038 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298687    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:20.305038 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298690    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:20.305038 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298693    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:20.305038 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298696    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:20.305038 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298699    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:20.305038 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298702    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:20.305038 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298706    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:20.305408 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298709    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:20.305408 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298712    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:20.305408 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298715    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:20.305408 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298718    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:20.305408 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298721    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:20.305408 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298723    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:20.305408 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298726    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:20.305408 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298728    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:20.305408 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298731    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:20.305408 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298734    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:20.305408 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298736    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:20.305408 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298739    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:20.305408 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298742    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:20.305408 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298745    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:20.305408 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298748    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:20.305408 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298750    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:20.305408 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298753    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:20.305408 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298756    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:20.305408 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298758    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:20.305408 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298761    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:20.305894 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298763    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:20.305894 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298766    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:20.305894 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298768    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:20.305894 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298771    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:20.305894 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298773    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:20.305894 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298776    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:20.305894 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298779    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:20.305894 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298781    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:20.305894 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298784    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:20.305894 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298787    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:20.305894 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298790    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:20.305894 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298793    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:20.305894 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298796    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:20.305894 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298798    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:20.305894 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298801    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:20.305894 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298803    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:20.305894 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298806    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:20.305894 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298809    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:20.305894 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298811    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:20.306433 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298814    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:20.306433 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298817    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:20.306433 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298819    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:20.306433 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298822    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:20.306433 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298824    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:20.306433 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298827    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:20.306433 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298830    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:20.306433 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298833    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:20.306433 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298835    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:20.306433 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298838    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:20.306433 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298841    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:20.306433 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298844    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:20.306433 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298846    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:20.306433 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298848    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:20.306433 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298851    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:20.306433 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298854    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:20.306433 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298856    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:20.306433 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298859    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:20.306433 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298862    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:20.306433 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298864    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:20.306919 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298867    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:20.306919 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298870    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:20.306919 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298873    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:20.306919 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298875    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:20.306919 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298878    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:20.306919 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298881    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:20.306919 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298884    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:20.306919 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298886    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:20.306919 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298889    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:20.306919 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298892    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:20.306919 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298894    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:20.306919 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298897    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:20.306919 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298899    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:20.306919 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298902    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:20.306919 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298906    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:20.306919 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298909    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:20.306919 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298912    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:20.306919 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298915    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:20.306919 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298918    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:20.306919 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298921    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:20.307480 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298923    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:20.307480 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298926    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:20.307480 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298929    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:20.307480 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298931    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:20.307480 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298934    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:20.307480 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298936    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:20.307480 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:20.298939    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:20.307480 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.298944    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 20:49:20.307480 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.299044    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 20:49:20.307480 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.300863    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 20:49:20.307480 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.301503    2566 server.go:1019] "Starting client certificate rotation"
Sep 17 20:49:20.307480 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.301597    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 20:49:20.307480 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.301638    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 20:49:20.307824 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.305052    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 20:49:20.307824 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.306158    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 20:49:20.313140 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.313123    2566 log.go:25] "Validated CRI v1 runtime API"
Sep 17 20:49:20.314643 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.314629    2566 log.go:25] "Validated CRI v1 image API"
Sep 17 20:49:20.315472 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.315455    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 20:49:20.317562 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.317542    2566 fs.go:135] Filesystem UUIDs: map[3a67e20c-4617-467f-a4b8-bb5e3ecd8eb3:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f0844c6d-b90f-4e35-9515-76ca38fbd6f3:/dev/nvme0n1p4]
Sep 17 20:49:20.317644 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.317559    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 20:49:20.321705 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.321574    2566 manager.go:216] Machine: {Timestamp:2026-09-17 20:49:20.32106387 +0000 UTC m=+1.931797092 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3178117 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fb7121ff524f70970b1ac6d08faf1 SystemUUID:ec2fb712-1ff5-24f7-0970-b1ac6d08faf1 BootID:e55f306c-bfc1-4658-8b00-2f97ec7963eb Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cb:77:96:6f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cb:77:96:6f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:4f:db:55:b0:eb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 20:49:20.321705 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.321696    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 20:49:20.321821 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.321769    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 20:49:20.323136 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.323109    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 20:49:20.323347 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.323142    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-140.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 20:49:20.324239 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.324229    2566 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 20:49:20.324273 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.324245    2566 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 20:49:20.324273 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.324253    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 20:49:20.324273 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.324262    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 20:49:20.324351 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.324327    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 20:49:20.327637 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.327626    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 20:49:20.327756 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.327746    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 20:49:20.328186 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.328177    2566 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 20:49:20.328231 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.328193    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 20:49:20.328231 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.328221    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 20:49:20.328231 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.328231    2566 kubelet.go:398] "Adding apiserver pod source"
Sep 17 20:49:20.328371 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.328241    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 20:49:20.328537 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.328527    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 20:49:20.328566 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.328545    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 20:49:20.330434 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.330413    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 20:49:20.332961 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.332936    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 20:49:20.333042 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.332972    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 20:49:20.334425 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.334404    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 20:49:20.334425 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.334432    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 20:49:20.334557 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.334443    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 20:49:20.334557 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.334451    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 20:49:20.334557 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.334463    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 20:49:20.334557 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.334471    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 20:49:20.334557 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.334490    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 20:49:20.334557 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.334498    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 20:49:20.334557 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.334508    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 20:49:20.334557 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.334517    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 20:49:20.334557 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.334540    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 20:49:20.334557 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.334548    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 20:49:20.334816 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.334577    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 20:49:20.334816 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.334586    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 20:49:20.337891 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.337878    2566 server.go:1268] "Started kubelet"
Sep 17 20:49:20.337982 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.337956    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 20:49:20.338143 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.338069    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 20:49:20.338143 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.338134    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 20:49:20.338793 ip-10-0-138-140 systemd[1]: Started Kubernetes Kubelet.
Sep 17 20:49:20.339878 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.339025    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 20:49:20.339878 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.339432    2566 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 20:49:20.339878 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.339745    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-140.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 20:49:20.340488 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.340470    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 20:49:20.340537 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:20.340505    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 20:49:20.340537 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:20.340505    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-140.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 20:49:20.343678 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:20.343662    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 20:49:20.344736 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.344719    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 20:49:20.344736 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.344725    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 20:49:20.344879 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:20.344784    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-140.ec2.internal\" not found"
Sep 17 20:49:20.345150 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.345133    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 20:49:20.345267 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.345146    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 20:49:20.345339 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.345292    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 20:49:20.345423 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.345409    2566 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 20:49:20.345423 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.345422    2566 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 20:49:20.345799 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.345784    2566 factory.go:153] Registering CRI-O factory
Sep 17 20:49:20.345799 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.345801    2566 factory.go:223] Registration of the crio container factory successfully
Sep 17 20:49:20.345926 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.345848    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 20:49:20.345926 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.345857    2566 factory.go:55] Registering systemd factory
Sep 17 20:49:20.345926 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.345864    2566 factory.go:223] Registration of the systemd container factory successfully
Sep 17 20:49:20.345926 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.345879    2566 factory.go:103] Registering Raw factory
Sep 17 20:49:20.345926 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.345891    2566 manager.go:1196] Started watching for new ooms in manager
Sep 17 20:49:20.346435 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.346412    2566 manager.go:349] Starting recovery of all containers
Sep 17 20:49:20.348203 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:20.348175    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 20:49:20.348316 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:20.348293    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-140.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 20:49:20.372159 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.372008    2566 manager.go:354] Recovery completed
Sep 17 20:49:20.377531 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.377516    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:20.379267 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.379247    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:20.379350 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.379286    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:20.379350 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.379301    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:20.379848 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.379836    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:20.379919 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.379840    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 20:49:20.379919 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.379868    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 20:49:20.379919 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.379885    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 20:49:20.380558 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.380542    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:20.380613 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.380568    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:20.380613 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.380578    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:20.381958 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.381947    2566 policy_none.go:49] "None policy: Start"
Sep 17 20:49:20.382002 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.381963    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 20:49:20.382002 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.381973    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 20:49:20.383882 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.383870    2566 policy_none.go:47] "Start"
Sep 17 20:49:20.388082 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:20.387352    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-140.ec2.internal.18d636ce01366261  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-140.ec2.internal,UID:ip-10-0-138-140.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-140.ec2.internal,},FirstTimestamp:2026-09-17 20:49:20.337855073 +0000 UTC m=+1.948588293,LastTimestamp:2026-09-17 20:49:20.337855073 +0000 UTC m=+1.948588293,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-140.ec2.internal,}"
Sep 17 20:49:20.396892 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.396876    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2tmql"
Sep 17 20:49:20.400194 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:20.400121    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-140.ec2.internal.18d636ce03ae47b8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-140.ec2.internal,UID:ip-10-0-138-140.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-140.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-140.ec2.internal,},FirstTimestamp:2026-09-17 20:49:20.379267 +0000 UTC m=+1.990000221,LastTimestamp:2026-09-17 20:49:20.379267 +0000 UTC m=+1.990000221,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-140.ec2.internal,}"
Sep 17 20:49:20.405200 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.405183    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2tmql"
Sep 17 20:49:20.410102 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:20.410041    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-140.ec2.internal.18d636ce03aeaafa  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-140.ec2.internal,UID:ip-10-0-138-140.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-140.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-140.ec2.internal,},FirstTimestamp:2026-09-17 20:49:20.37929241 +0000 UTC m=+1.990025632,LastTimestamp:2026-09-17 20:49:20.37929241 +0000 UTC m=+1.990025632,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-140.ec2.internal,}"
Sep 17 20:49:20.417823 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.417809    2566 manager.go:337] "Starting Device Plugin manager"
Sep 17 20:49:20.439439 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:20.417842    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 20:49:20.439439 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.417854    2566 server.go:87] "Starting device plugin registration server"
Sep 17 20:49:20.439439 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.418076    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 20:49:20.439439 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.418093    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 20:49:20.439439 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.418175    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 20:49:20.439439 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.418279    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 20:49:20.439439 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.418288    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 20:49:20.439439 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:20.418814    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 20:49:20.439439 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:20.418853    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-140.ec2.internal\" not found"
Sep 17 20:49:20.452604 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.452570    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 20:49:20.453706 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.453686    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 20:49:20.453775 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.453730    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 20:49:20.453775 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.453752    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 20:49:20.453852 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:20.453785    2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 20:49:20.455641 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.455621    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:20.518819 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.518800    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:20.519598 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.519582    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:20.519671 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.519612    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:20.519671 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.519621    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:20.519671 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.519642    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-140.ec2.internal"
Sep 17 20:49:20.527597 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.527579    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-140.ec2.internal"
Sep 17 20:49:20.527643 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:20.527604    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-140.ec2.internal\": node \"ip-10-0-138-140.ec2.internal\" not found"
Sep 17 20:49:20.544962 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:20.544920    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-140.ec2.internal\" not found"
Sep 17 20:49:20.554449 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.554429    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-140.ec2.internal"]
Sep 17 20:49:20.554495 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.554490    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:20.555180 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.555166    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:20.555254 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.555187    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:20.555254 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.555224    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:20.557385 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.557374    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:20.557509 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.557495    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal"
Sep 17 20:49:20.557547 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.557526    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:20.558026 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.558010    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:20.558119 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.558040    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:20.558119 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.558054    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:20.558119 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.558011    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:20.558119 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.558110    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:20.558332 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.558123    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:20.560297 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.560283    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-140.ec2.internal"
Sep 17 20:49:20.560361 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.560307    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:20.560886 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.560871    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:20.560969 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.560894    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:20.560969 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.560908    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:20.575831 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:20.575814    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-140.ec2.internal\" not found" node="ip-10-0-138-140.ec2.internal"
Sep 17 20:49:20.579507 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:20.579494    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-140.ec2.internal\" not found" node="ip-10-0-138-140.ec2.internal"
Sep 17 20:49:20.645079 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:20.645061    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-140.ec2.internal\" not found"
Sep 17 20:49:20.647343 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.647329    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e97c66319ae520682604e45d82c8b847-config\") pod \"kube-apiserver-proxy-ip-10-0-138-140.ec2.internal\" (UID: \"e97c66319ae520682604e45d82c8b847\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-140.ec2.internal"
Sep 17 20:49:20.647390 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.647353    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cce51aae58d840086c38254c622dda75-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal\" (UID: \"cce51aae58d840086c38254c622dda75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal"
Sep 17 20:49:20.647390 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.647384    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cce51aae58d840086c38254c622dda75-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal\" (UID: \"cce51aae58d840086c38254c622dda75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal"
Sep 17 20:49:20.745683 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:20.745661    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-140.ec2.internal\" not found"
Sep 17 20:49:20.747863 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.747848    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cce51aae58d840086c38254c622dda75-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal\" (UID: \"cce51aae58d840086c38254c622dda75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal"
Sep 17 20:49:20.747915 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.747871    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cce51aae58d840086c38254c622dda75-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal\" (UID: \"cce51aae58d840086c38254c622dda75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal"
Sep 17 20:49:20.747915 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.747890    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e97c66319ae520682604e45d82c8b847-config\") pod \"kube-apiserver-proxy-ip-10-0-138-140.ec2.internal\" (UID: \"e97c66319ae520682604e45d82c8b847\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-140.ec2.internal"
Sep 17 20:49:20.748003 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.747914    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e97c66319ae520682604e45d82c8b847-config\") pod \"kube-apiserver-proxy-ip-10-0-138-140.ec2.internal\" (UID: \"e97c66319ae520682604e45d82c8b847\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-140.ec2.internal"
Sep 17 20:49:20.748003 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.747937    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cce51aae58d840086c38254c622dda75-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal\" (UID: \"cce51aae58d840086c38254c622dda75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal"
Sep 17 20:49:20.748003 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.747955    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cce51aae58d840086c38254c622dda75-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal\" (UID: \"cce51aae58d840086c38254c622dda75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal"
Sep 17 20:49:20.846366 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:20.846290    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-140.ec2.internal\" not found"
Sep 17 20:49:20.878602 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.878580    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal"
Sep 17 20:49:20.882260 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:20.882238    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-140.ec2.internal"
Sep 17 20:49:20.947429 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:20.947405    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-140.ec2.internal\" not found"
Sep 17 20:49:21.047815 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:21.047795    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-140.ec2.internal\" not found"
Sep 17 20:49:21.148318 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:21.148260    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-140.ec2.internal\" not found"
Sep 17 20:49:21.248688 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:21.248660    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-140.ec2.internal\" not found"
Sep 17 20:49:21.301916 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:21.301892    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 20:49:21.302437 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:21.302030    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 20:49:21.349186 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:21.349163    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-140.ec2.internal\" not found"
Sep 17 20:49:21.410067 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:21.409985    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 20:44:20 +0000 UTC" deadline="2028-05-11 01:56:47.511536594 +0000 UTC"
Sep 17 20:49:21.410067 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:21.410042    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14429h7m26.101504421s"
Sep 17 20:49:21.449494 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:21.449472    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-140.ec2.internal\" not found"
Sep 17 20:49:21.521263 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:21.521244    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:21.537645 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:21.537605    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce51aae58d840086c38254c622dda75.slice/crio-f444bce8421987d79ac9b9cd9a8a5012937b9216e237000a73ff392cb231b08d WatchSource:0}: Error finding container f444bce8421987d79ac9b9cd9a8a5012937b9216e237000a73ff392cb231b08d: Status 404 returned error can't find the container with id f444bce8421987d79ac9b9cd9a8a5012937b9216e237000a73ff392cb231b08d
Sep 17 20:49:21.538419 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:21.538395    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode97c66319ae520682604e45d82c8b847.slice/crio-5a67ddf8dcc7767d4c7e8993cc29829517ddf91f84dea36bd1efe93f346679c0 WatchSource:0}: Error finding container 5a67ddf8dcc7767d4c7e8993cc29829517ddf91f84dea36bd1efe93f346679c0: Status 404 returned error can't find the container with id 5a67ddf8dcc7767d4c7e8993cc29829517ddf91f84dea36bd1efe93f346679c0
Sep 17 20:49:21.543732 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:21.543718    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 20:49:21.545406 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:21.545389    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:21.546247 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:21.546232    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:21.546334 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:21.546261    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:21.546334 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:21.546281    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:21.550308 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:21.550293    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-140.ec2.internal\" not found"
Sep 17 20:49:21.651246 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:21.651222    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-140.ec2.internal\" not found"
Sep 17 20:49:21.728889 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:21.728672    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:21.752006 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:21.751987    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-140.ec2.internal\" not found"
Sep 17 20:49:21.852671 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:21.852645    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-140.ec2.internal\" not found"
Sep 17 20:49:21.935190 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:21.935159    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:21.945710 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:21.945543    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal"
Sep 17 20:49:21.959250 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:21.959224    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 20:49:21.960068 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:21.960050    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-140.ec2.internal"
Sep 17 20:49:21.967825 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:21.967750    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 20:49:22.330449 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.330422    2566 apiserver.go:52] "Watching apiserver"
Sep 17 20:49:22.338859 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.338837    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 20:49:22.339177 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.339158    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-9tkmw","openshift-network-operator/iptables-alerter-4nh74","openshift-ovn-kubernetes/ovnkube-node-bnqx7","kube-system/konnectivity-agent-6mj5z","kube-system/kube-apiserver-proxy-ip-10-0-138-140.ec2.internal","openshift-cluster-node-tuning-operator/tuned-bzvzr","openshift-dns/node-resolver-j7t6x","openshift-image-registry/node-ca-j8n57","openshift-multus/multus-additional-cni-plugins-rw72g","openshift-multus/multus-h5f4t","openshift-multus/network-metrics-daemon-sknnf","kube-system/global-pull-secret-syncer-ltk4w","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n9vj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal"]
Sep 17 20:49:22.342378 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.342236    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j7t6x"
Sep 17 20:49:22.344804 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.344776    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:49:22.344905 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:22.344852    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sknnf" podUID="8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7"
Sep 17 20:49:22.345631 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.345604    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2xkdc\""
Sep 17 20:49:22.346012 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.345992    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 20:49:22.346103 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.346045    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 20:49:22.346103 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.346045    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 20:49:22.347165 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.347142    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.349373 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.349355    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 20:49:22.349607 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.349591    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6mj5z"
Sep 17 20:49:22.351123 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.350580    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 20:49:22.351123 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.350616    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 20:49:22.351123 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.350753    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 20:49:22.351123 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.350781    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 20:49:22.351123 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.350919    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 20:49:22.351495 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.351147    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s4cm7\""
Sep 17 20:49:22.351735 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.351717    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j8n57"
Sep 17 20:49:22.352645 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.352630    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 20:49:22.355582 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.354499    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 20:49:22.355582 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.354542    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 20:49:22.355582 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.354682    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 20:49:22.355582 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.354708    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-76sgp\""
Sep 17 20:49:22.355582 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.354852    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 20:49:22.355582 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.354932    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sjxdf\""
Sep 17 20:49:22.355582 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.355260    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.355947 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.355686    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/402f55a4-24b6-47c6-b719-9c6a5b473434-env-overrides\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.355947 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.355725    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg5dd\" (UniqueName: \"kubernetes.io/projected/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-kube-api-access-mg5dd\") pod \"network-metrics-daemon-sknnf\" (UID: \"8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7\") " pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:49:22.355947 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.355752    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-node-log\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.355947 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.355804    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-log-socket\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.355947 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.355838    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-host-run-ovn-kubernetes\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.355947 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.355876    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/402f55a4-24b6-47c6-b719-9c6a5b473434-ovn-node-metrics-cert\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.355947 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.355900    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/402f55a4-24b6-47c6-b719-9c6a5b473434-ovnkube-script-lib\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.355947 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.355924    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtfbh\" (UniqueName: \"kubernetes.io/projected/5f5d4daa-933c-47d6-b93a-6f83ed7ab23f-kube-api-access-rtfbh\") pod \"node-resolver-j7t6x\" (UID: \"5f5d4daa-933c-47d6-b93a-6f83ed7ab23f\") " pod="openshift-dns/node-resolver-j7t6x"
Sep 17 20:49:22.355947 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.355945    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-host-kubelet\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.356341 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.355968    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-host-slash\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.356341 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.355989    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-run-systemd\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.356341 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.356010    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-etc-openvswitch\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.356341 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.356034    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-run-openvswitch\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.356341 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.356058    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-host-cni-bin\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.356341 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.356080    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/402f55a4-24b6-47c6-b719-9c6a5b473434-ovnkube-config\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.356341 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.356102    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5f5d4daa-933c-47d6-b93a-6f83ed7ab23f-hosts-file\") pod \"node-resolver-j7t6x\" (UID: \"5f5d4daa-933c-47d6-b93a-6f83ed7ab23f\") " pod="openshift-dns/node-resolver-j7t6x"
Sep 17 20:49:22.356341 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.356124    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-host-run-netns\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.356341 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.356149    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-run-ovn\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.356341 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.356183    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qngp\" (UniqueName: \"kubernetes.io/projected/402f55a4-24b6-47c6-b719-9c6a5b473434-kube-api-access-8qngp\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.356341 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.356246    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-var-lib-openvswitch\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.356341 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.356279    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f5d4daa-933c-47d6-b93a-6f83ed7ab23f-tmp-dir\") pod \"node-resolver-j7t6x\" (UID: \"5f5d4daa-933c-47d6-b93a-6f83ed7ab23f\") " pod="openshift-dns/node-resolver-j7t6x"
Sep 17 20:49:22.356341 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.356320    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-metrics-certs\") pod \"network-metrics-daemon-sknnf\" (UID: \"8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7\") " pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:49:22.356734 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.356346    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-systemd-units\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.356734 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.356371    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-host-cni-netd\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.356734 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.356401    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.357891 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.357872    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rw72g"
Sep 17 20:49:22.360379 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.360122    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 20:49:22.360867 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.360640    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tmm7f\""
Sep 17 20:49:22.360867 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.360665    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n9vj"
Sep 17 20:49:22.363064 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.362829    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fd4tf\""
Sep 17 20:49:22.363064 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.362883    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 20:49:22.363064 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.362933    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 20:49:22.363064 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.362993    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 20:49:22.363342 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.363192    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:49:22.363342 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.363308    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 20:49:22.363528 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.363505    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 20:49:22.364922 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.363698    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4nh74"
Sep 17 20:49:22.365690 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.365670    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 20:49:22.365951 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.365672    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 20:49:22.366781 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.366762    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 20:49:22.367030 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.367011    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 20:49:22.367098 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.367067    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:49:22.367392 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.367373    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.368442 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.368428    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 20:49:22.368442 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.368436    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v4x6q\""
Sep 17 20:49:22.368556 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.368542    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f78qn\""
Sep 17 20:49:22.369640 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.369610    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 20:49:22.369640 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.369616    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tkmw"
Sep 17 20:49:22.369797 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:22.369702    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tkmw" podUID="3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b"
Sep 17 20:49:22.370037 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.370021    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-86lvv\""
Sep 17 20:49:22.372150 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.372132    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:22.372248 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:22.372190    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltk4w" podUID="3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a"
Sep 17 20:49:22.372870 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.372851    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 20:49:22.446219 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.446175    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 20:49:22.446751 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.446732    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-j25tp"
Sep 17 20:49:22.456598 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.456575    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-run-ovn\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.456701 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.456608    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67a00ce8-eae6-48cb-896a-bb6de23904e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rw72g\" (UID: \"67a00ce8-eae6-48cb-896a-bb6de23904e9\") " pod="openshift-multus/multus-additional-cni-plugins-rw72g"
Sep 17 20:49:22.456701 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.456626    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-host-var-lib-kubelet\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.456701 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.456647    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5e2bd987-6d8a-457b-ad67-f1e6597864e5-serviceca\") pod \"node-ca-j8n57\" (UID: \"5e2bd987-6d8a-457b-ad67-f1e6597864e5\") " pod="openshift-image-registry/node-ca-j8n57"
Sep 17 20:49:22.456701 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.456669    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzrc6\" (UniqueName: \"kubernetes.io/projected/5e2bd987-6d8a-457b-ad67-f1e6597864e5-kube-api-access-pzrc6\") pod \"node-ca-j8n57\" (UID: \"5e2bd987-6d8a-457b-ad67-f1e6597864e5\") " pod="openshift-image-registry/node-ca-j8n57"
Sep 17 20:49:22.456886 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.456716    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-var-lib-openvswitch\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.456886 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.456773    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-var-lib-openvswitch\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.456886 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.456786    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-run-ovn\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.456886 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.456778    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-systemd-units\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.456886 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.456826    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67a00ce8-eae6-48cb-896a-bb6de23904e9-cnibin\") pod \"multus-additional-cni-plugins-rw72g\" (UID: \"67a00ce8-eae6-48cb-896a-bb6de23904e9\") " pod="openshift-multus/multus-additional-cni-plugins-rw72g"
Sep 17 20:49:22.456886 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.456843    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-host-var-lib-cni-multus\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.456886 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.456865    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j6sf\" (UniqueName: \"kubernetes.io/projected/3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b-kube-api-access-2j6sf\") pod \"network-check-target-9tkmw\" (UID: \"3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b\") " pod="openshift-network-diagnostics/network-check-target-9tkmw"
Sep 17 20:49:22.456886 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.456879    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-systemd-units\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.456886 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.456889    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-log-socket\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.457319 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.456919    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-log-socket\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.457319 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.456950    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/402f55a4-24b6-47c6-b719-9c6a5b473434-ovn-node-metrics-cert\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.457319 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.456977    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-cnibin\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.457319 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.457028    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-multus-conf-dir\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.457319 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.457051    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a-kubelet-config\") pod \"global-pull-secret-syncer-ltk4w\" (UID: \"3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a\") " pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:22.457319 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.457243    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtfbh\" (UniqueName: \"kubernetes.io/projected/5f5d4daa-933c-47d6-b93a-6f83ed7ab23f-kube-api-access-rtfbh\") pod \"node-resolver-j7t6x\" (UID: \"5f5d4daa-933c-47d6-b93a-6f83ed7ab23f\") " pod="openshift-dns/node-resolver-j7t6x"
Sep 17 20:49:22.457319 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.457274    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-run-openvswitch\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.457319 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.457299    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-host-cni-bin\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.457319 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.457317    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qngp\" (UniqueName: \"kubernetes.io/projected/402f55a4-24b6-47c6-b719-9c6a5b473434-kube-api-access-8qngp\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.457729 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.457343    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-etc-sysctl-d\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.457729 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.457363    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-etc-sysctl-conf\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.457729 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.457389    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/67a00ce8-eae6-48cb-896a-bb6de23904e9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rw72g\" (UID: \"67a00ce8-eae6-48cb-896a-bb6de23904e9\") " pod="openshift-multus/multus-additional-cni-plugins-rw72g"
Sep 17 20:49:22.457729 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.457418    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhb2k\" (UniqueName: \"kubernetes.io/projected/67a00ce8-eae6-48cb-896a-bb6de23904e9-kube-api-access-dhb2k\") pod \"multus-additional-cni-plugins-rw72g\" (UID: \"67a00ce8-eae6-48cb-896a-bb6de23904e9\") " pod="openshift-multus/multus-additional-cni-plugins-rw72g"
Sep 17 20:49:22.457729 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.457421    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-host-cni-bin\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.457729 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.457440    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-etc-systemd\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.457729 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.457464    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce8d6a4d-e013-4e57-a540-14ddc06d7745-tmp\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.457729 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.457471    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-run-openvswitch\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.457729 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.457472    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 20:49:22.457729 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.457491    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5caabfba-6743-414e-8d96-caf028817a3b-multus-daemon-config\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.457729 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.457621    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-etc-kubernetes\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.457729 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.457645    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21362652-a68e-4d69-8cea-69002798058b-host-slash\") pod \"iptables-alerter-4nh74\" (UID: \"21362652-a68e-4d69-8cea-69002798058b\") " pod="openshift-network-operator/iptables-alerter-4nh74"
Sep 17 20:49:22.457729 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.457674    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-metrics-certs\") pod \"network-metrics-daemon-sknnf\" (UID: \"8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7\") " pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:49:22.457729 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.457696    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d-device-dir\") pod \"aws-ebs-csi-driver-node-4n9vj\" (UID: \"f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n9vj"
Sep 17 20:49:22.457729 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.457731    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-host-var-lib-cni-bin\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.458476 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:22.457811    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:22.458476 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.457844    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg5dd\" (UniqueName: \"kubernetes.io/projected/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-kube-api-access-mg5dd\") pod \"network-metrics-daemon-sknnf\" (UID: \"8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7\") " pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:49:22.458476 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.457872    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/582bf447-0797-4411-bce3-89e1c2455f6d-agent-certs\") pod \"konnectivity-agent-6mj5z\" (UID: \"582bf447-0797-4411-bce3-89e1c2455f6d\") " pod="kube-system/konnectivity-agent-6mj5z"
Sep 17 20:49:22.458476 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:22.457923    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-metrics-certs podName:8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:22.957883243 +0000 UTC m=+4.568616480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-metrics-certs") pod "network-metrics-daemon-sknnf" (UID: "8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:22.458476 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.457978    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/67a00ce8-eae6-48cb-896a-bb6de23904e9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rw72g\" (UID: \"67a00ce8-eae6-48cb-896a-bb6de23904e9\") " pod="openshift-multus/multus-additional-cni-plugins-rw72g"
Sep 17 20:49:22.458476 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458009    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d-registration-dir\") pod \"aws-ebs-csi-driver-node-4n9vj\" (UID: \"f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n9vj"
Sep 17 20:49:22.458476 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458037    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/402f55a4-24b6-47c6-b719-9c6a5b473434-ovnkube-config\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.458476 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458060    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-etc-kubernetes\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.458476 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458082    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ce8d6a4d-e013-4e57-a540-14ddc06d7745-etc-tuned\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.458476 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458104    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67a00ce8-eae6-48cb-896a-bb6de23904e9-system-cni-dir\") pod \"multus-additional-cni-plugins-rw72g\" (UID: \"67a00ce8-eae6-48cb-896a-bb6de23904e9\") " pod="openshift-multus/multus-additional-cni-plugins-rw72g"
Sep 17 20:49:22.458476 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458127    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-host-run-netns\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.458476 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458148    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/21362652-a68e-4d69-8cea-69002798058b-iptables-alerter-script\") pod \"iptables-alerter-4nh74\" (UID: \"21362652-a68e-4d69-8cea-69002798058b\") " pod="openshift-network-operator/iptables-alerter-4nh74"
Sep 17 20:49:22.458476 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458170    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a-dbus\") pod \"global-pull-secret-syncer-ltk4w\" (UID: \"3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a\") " pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:22.458476 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458194    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-host-cni-netd\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.458476 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458232    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/402f55a4-24b6-47c6-b719-9c6a5b473434-env-overrides\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.458476 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458258    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67a00ce8-eae6-48cb-896a-bb6de23904e9-cni-binary-copy\") pod \"multus-additional-cni-plugins-rw72g\" (UID: \"67a00ce8-eae6-48cb-896a-bb6de23904e9\") " pod="openshift-multus/multus-additional-cni-plugins-rw72g"
Sep 17 20:49:22.459194 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458284    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfkpc\" (UniqueName: \"kubernetes.io/projected/f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d-kube-api-access-lfkpc\") pod \"aws-ebs-csi-driver-node-4n9vj\" (UID: \"f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n9vj"
Sep 17 20:49:22.459194 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458306    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5caabfba-6743-414e-8d96-caf028817a3b-cni-binary-copy\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.459194 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458337    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a-original-pull-secret\") pod \"global-pull-secret-syncer-ltk4w\" (UID: \"3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a\") " pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:22.459194 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458305    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-host-cni-netd\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.459194 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458375    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-etc-modprobe-d\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.459194 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458407    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-sys\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.459194 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458422    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d-etc-selinux\") pod \"aws-ebs-csi-driver-node-4n9vj\" (UID: \"f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n9vj"
Sep 17 20:49:22.459194 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458440    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-system-cni-dir\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.459194 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458479    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-run-systemd\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.459194 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458514    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-etc-openvswitch\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.459194 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458542    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-run\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.459194 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458548    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-run-systemd\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.459194 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458562    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-etc-openvswitch\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.459194 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458567    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-var-lib-kubelet\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.459194 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458592    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d-socket-dir\") pod \"aws-ebs-csi-driver-node-4n9vj\" (UID: \"f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n9vj"
Sep 17 20:49:22.459194 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458607    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-multus-socket-dir-parent\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.459194 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458628    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-etc-sysconfig\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.459979 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458644    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4n9vj\" (UID: \"f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n9vj"
Sep 17 20:49:22.459979 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458626    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/402f55a4-24b6-47c6-b719-9c6a5b473434-env-overrides\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.459979 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458665    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd24m\" (UniqueName: \"kubernetes.io/projected/5caabfba-6743-414e-8d96-caf028817a3b-kube-api-access-bd24m\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.459979 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458649    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/402f55a4-24b6-47c6-b719-9c6a5b473434-ovnkube-config\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.459979 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458713    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e2bd987-6d8a-457b-ad67-f1e6597864e5-host\") pod \"node-ca-j8n57\" (UID: \"5e2bd987-6d8a-457b-ad67-f1e6597864e5\") " pod="openshift-image-registry/node-ca-j8n57"
Sep 17 20:49:22.459979 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458745    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkq6j\" (UniqueName: \"kubernetes.io/projected/21362652-a68e-4d69-8cea-69002798058b-kube-api-access-tkq6j\") pod \"iptables-alerter-4nh74\" (UID: \"21362652-a68e-4d69-8cea-69002798058b\") " pod="openshift-network-operator/iptables-alerter-4nh74"
Sep 17 20:49:22.459979 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458807    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f5d4daa-933c-47d6-b93a-6f83ed7ab23f-tmp-dir\") pod \"node-resolver-j7t6x\" (UID: \"5f5d4daa-933c-47d6-b93a-6f83ed7ab23f\") " pod="openshift-dns/node-resolver-j7t6x"
Sep 17 20:49:22.459979 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458841    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.459979 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458863    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-lib-modules\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.459979 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458883    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4l7m\" (UniqueName: \"kubernetes.io/projected/ce8d6a4d-e013-4e57-a540-14ddc06d7745-kube-api-access-s4l7m\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.459979 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458898    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-multus-cni-dir\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.459979 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458914    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-node-log\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.459979 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458935    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-host-run-ovn-kubernetes\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.459979 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458962    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/402f55a4-24b6-47c6-b719-9c6a5b473434-ovnkube-script-lib\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.459979 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.458985    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/582bf447-0797-4411-bce3-89e1c2455f6d-konnectivity-ca\") pod \"konnectivity-agent-6mj5z\" (UID: \"582bf447-0797-4411-bce3-89e1c2455f6d\") " pod="kube-system/konnectivity-agent-6mj5z"
Sep 17 20:49:22.459979 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.459009    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-host\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.459979 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.459046    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-host-kubelet\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.460670 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.459070    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-host-run-ovn-kubernetes\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.460670 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.459091    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-host-slash\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.460670 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.459101    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f5d4daa-933c-47d6-b93a-6f83ed7ab23f-tmp-dir\") pod \"node-resolver-j7t6x\" (UID: \"5f5d4daa-933c-47d6-b93a-6f83ed7ab23f\") " pod="openshift-dns/node-resolver-j7t6x"
Sep 17 20:49:22.460670 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.459118    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d-sys-fs\") pod \"aws-ebs-csi-driver-node-4n9vj\" (UID: \"f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n9vj"
Sep 17 20:49:22.460670 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.459145    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-host-run-k8s-cni-cncf-io\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.460670 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.459168    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-hostroot\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.460670 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.459192    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-host-run-multus-certs\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.460670 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.459263    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5f5d4daa-933c-47d6-b93a-6f83ed7ab23f-hosts-file\") pod \"node-resolver-j7t6x\" (UID: \"5f5d4daa-933c-47d6-b93a-6f83ed7ab23f\") " pod="openshift-dns/node-resolver-j7t6x"
Sep 17 20:49:22.460670 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.459289    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-host-run-netns\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.460670 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.459307    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.460670 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.459368    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-host-kubelet\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.460670 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.459383    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-host-run-netns\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.460670 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.459437    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-host-slash\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.460670 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.459051    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/402f55a4-24b6-47c6-b719-9c6a5b473434-node-log\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.460670 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.459500    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5f5d4daa-933c-47d6-b93a-6f83ed7ab23f-hosts-file\") pod \"node-resolver-j7t6x\" (UID: \"5f5d4daa-933c-47d6-b93a-6f83ed7ab23f\") " pod="openshift-dns/node-resolver-j7t6x"
Sep 17 20:49:22.460670 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.459515    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/402f55a4-24b6-47c6-b719-9c6a5b473434-ovnkube-script-lib\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.460670 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.459729    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal" event={"ID":"cce51aae58d840086c38254c622dda75","Type":"ContainerStarted","Data":"f444bce8421987d79ac9b9cd9a8a5012937b9216e237000a73ff392cb231b08d"}
Sep 17 20:49:22.461373 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.461120    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-140.ec2.internal" event={"ID":"e97c66319ae520682604e45d82c8b847","Type":"ContainerStarted","Data":"5a67ddf8dcc7767d4c7e8993cc29829517ddf91f84dea36bd1efe93f346679c0"}
Sep 17 20:49:22.461533 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.461513    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/402f55a4-24b6-47c6-b719-9c6a5b473434-ovn-node-metrics-cert\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.485583 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.485559    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-j25tp"
Sep 17 20:49:22.488016 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.487988    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtfbh\" (UniqueName: \"kubernetes.io/projected/5f5d4daa-933c-47d6-b93a-6f83ed7ab23f-kube-api-access-rtfbh\") pod \"node-resolver-j7t6x\" (UID: \"5f5d4daa-933c-47d6-b93a-6f83ed7ab23f\") " pod="openshift-dns/node-resolver-j7t6x"
Sep 17 20:49:22.488104 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.488065    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qngp\" (UniqueName: \"kubernetes.io/projected/402f55a4-24b6-47c6-b719-9c6a5b473434-kube-api-access-8qngp\") pod \"ovnkube-node-bnqx7\" (UID: \"402f55a4-24b6-47c6-b719-9c6a5b473434\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.489526 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.489504    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg5dd\" (UniqueName: \"kubernetes.io/projected/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-kube-api-access-mg5dd\") pod \"network-metrics-daemon-sknnf\" (UID: \"8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7\") " pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:49:22.560005 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.559977    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-cnibin\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.560103 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.560009    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-multus-conf-dir\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.561319 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.560361    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-cnibin\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.561319 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.560371    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-multus-conf-dir\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.561319 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.560472    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a-kubelet-config\") pod \"global-pull-secret-syncer-ltk4w\" (UID: \"3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a\") " pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:22.561319 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.560521    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a-kubelet-config\") pod \"global-pull-secret-syncer-ltk4w\" (UID: \"3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a\") " pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:22.561319 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.560548    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-etc-sysctl-d\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.561319 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.560599    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-etc-sysctl-conf\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.561319 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.560717    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/67a00ce8-eae6-48cb-896a-bb6de23904e9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rw72g\" (UID: \"67a00ce8-eae6-48cb-896a-bb6de23904e9\") " pod="openshift-multus/multus-additional-cni-plugins-rw72g"
Sep 17 20:49:22.561319 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.560752    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhb2k\" (UniqueName: \"kubernetes.io/projected/67a00ce8-eae6-48cb-896a-bb6de23904e9-kube-api-access-dhb2k\") pod \"multus-additional-cni-plugins-rw72g\" (UID: \"67a00ce8-eae6-48cb-896a-bb6de23904e9\") " pod="openshift-multus/multus-additional-cni-plugins-rw72g"
Sep 17 20:49:22.561319 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.560788    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-etc-sysctl-d\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.561319 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.560782    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-etc-systemd\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.561319 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.560848    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce8d6a4d-e013-4e57-a540-14ddc06d7745-tmp\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.561319 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.560860    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-etc-systemd\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.561319 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.560878    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5caabfba-6743-414e-8d96-caf028817a3b-multus-daemon-config\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.561319 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.560913    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-etc-kubernetes\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.561319 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.560950    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21362652-a68e-4d69-8cea-69002798058b-host-slash\") pod \"iptables-alerter-4nh74\" (UID: \"21362652-a68e-4d69-8cea-69002798058b\") " pod="openshift-network-operator/iptables-alerter-4nh74"
Sep 17 20:49:22.561319 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.560998    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d-device-dir\") pod \"aws-ebs-csi-driver-node-4n9vj\" (UID: \"f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n9vj"
Sep 17 20:49:22.561319 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.561030    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-host-var-lib-cni-bin\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.561989 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.561070    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/582bf447-0797-4411-bce3-89e1c2455f6d-agent-certs\") pod \"konnectivity-agent-6mj5z\" (UID: \"582bf447-0797-4411-bce3-89e1c2455f6d\") " pod="kube-system/konnectivity-agent-6mj5z"
Sep 17 20:49:22.561989 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.561115    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/67a00ce8-eae6-48cb-896a-bb6de23904e9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rw72g\" (UID: \"67a00ce8-eae6-48cb-896a-bb6de23904e9\") " pod="openshift-multus/multus-additional-cni-plugins-rw72g"
Sep 17 20:49:22.561989 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.561144    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d-registration-dir\") pod \"aws-ebs-csi-driver-node-4n9vj\" (UID: \"f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n9vj"
Sep 17 20:49:22.561989 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.561165    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-etc-sysctl-conf\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.561989 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.561176    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-etc-kubernetes\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.561989 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.561229    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ce8d6a4d-e013-4e57-a540-14ddc06d7745-etc-tuned\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.561989 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.561268    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67a00ce8-eae6-48cb-896a-bb6de23904e9-system-cni-dir\") pod \"multus-additional-cni-plugins-rw72g\" (UID: \"67a00ce8-eae6-48cb-896a-bb6de23904e9\") " pod="openshift-multus/multus-additional-cni-plugins-rw72g"
Sep 17 20:49:22.561989 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.561285    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d-device-dir\") pod \"aws-ebs-csi-driver-node-4n9vj\" (UID: \"f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n9vj"
Sep 17 20:49:22.561989 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.561305    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-host-run-netns\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.561989 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.561319    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/67a00ce8-eae6-48cb-896a-bb6de23904e9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rw72g\" (UID: \"67a00ce8-eae6-48cb-896a-bb6de23904e9\") " pod="openshift-multus/multus-additional-cni-plugins-rw72g"
Sep 17 20:49:22.561989 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.561333    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/21362652-a68e-4d69-8cea-69002798058b-iptables-alerter-script\") pod \"iptables-alerter-4nh74\" (UID: \"21362652-a68e-4d69-8cea-69002798058b\") " pod="openshift-network-operator/iptables-alerter-4nh74"
Sep 17 20:49:22.561989 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.561390    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21362652-a68e-4d69-8cea-69002798058b-host-slash\") pod \"iptables-alerter-4nh74\" (UID: \"21362652-a68e-4d69-8cea-69002798058b\") " pod="openshift-network-operator/iptables-alerter-4nh74"
Sep 17 20:49:22.561989 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.561434    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a-dbus\") pod \"global-pull-secret-syncer-ltk4w\" (UID: \"3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a\") " pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:22.561989 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.561493    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67a00ce8-eae6-48cb-896a-bb6de23904e9-cni-binary-copy\") pod \"multus-additional-cni-plugins-rw72g\" (UID: \"67a00ce8-eae6-48cb-896a-bb6de23904e9\") " pod="openshift-multus/multus-additional-cni-plugins-rw72g"
Sep 17 20:49:22.561989 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.561534    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfkpc\" (UniqueName: \"kubernetes.io/projected/f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d-kube-api-access-lfkpc\") pod \"aws-ebs-csi-driver-node-4n9vj\" (UID: \"f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n9vj"
Sep 17 20:49:22.561989 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.561584    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5caabfba-6743-414e-8d96-caf028817a3b-cni-binary-copy\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.561989 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.561634    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a-original-pull-secret\") pod \"global-pull-secret-syncer-ltk4w\" (UID: \"3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a\") " pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:22.562687 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.561672    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-etc-modprobe-d\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.562687 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.561707    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-sys\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.562687 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.561756    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d-etc-selinux\") pod \"aws-ebs-csi-driver-node-4n9vj\" (UID: \"f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n9vj"
Sep 17 20:49:22.562687 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.561783    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-system-cni-dir\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.562687 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.561816    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-run\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.562687 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.561844    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-var-lib-kubelet\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.562687 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.561890    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d-socket-dir\") pod \"aws-ebs-csi-driver-node-4n9vj\" (UID: \"f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n9vj"
Sep 17 20:49:22.562687 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.561918    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-etc-kubernetes\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.562687 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.561930    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-multus-socket-dir-parent\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.562687 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.561953    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-etc-sysconfig\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.562687 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.561993    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4n9vj\" (UID: \"f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n9vj"
Sep 17 20:49:22.562687 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.562019    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd24m\" (UniqueName: \"kubernetes.io/projected/5caabfba-6743-414e-8d96-caf028817a3b-kube-api-access-bd24m\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.562687 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.562044    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e2bd987-6d8a-457b-ad67-f1e6597864e5-host\") pod \"node-ca-j8n57\" (UID: \"5e2bd987-6d8a-457b-ad67-f1e6597864e5\") " pod="openshift-image-registry/node-ca-j8n57"
Sep 17 20:49:22.562687 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.562080    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-etc-modprobe-d\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.562687 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.562085    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkq6j\" (UniqueName: \"kubernetes.io/projected/21362652-a68e-4d69-8cea-69002798058b-kube-api-access-tkq6j\") pod \"iptables-alerter-4nh74\" (UID: \"21362652-a68e-4d69-8cea-69002798058b\") " pod="openshift-network-operator/iptables-alerter-4nh74"
Sep 17 20:49:22.562687 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.562165    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-lib-modules\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.562687 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.562193    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4l7m\" (UniqueName: \"kubernetes.io/projected/ce8d6a4d-e013-4e57-a540-14ddc06d7745-kube-api-access-s4l7m\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.563326 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.562362    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/21362652-a68e-4d69-8cea-69002798058b-iptables-alerter-script\") pod \"iptables-alerter-4nh74\" (UID: \"21362652-a68e-4d69-8cea-69002798058b\") " pod="openshift-network-operator/iptables-alerter-4nh74"
Sep 17 20:49:22.563326 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.562363    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-multus-cni-dir\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.563326 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.562445    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/582bf447-0797-4411-bce3-89e1c2455f6d-konnectivity-ca\") pod \"konnectivity-agent-6mj5z\" (UID: \"582bf447-0797-4411-bce3-89e1c2455f6d\") " pod="kube-system/konnectivity-agent-6mj5z"
Sep 17 20:49:22.563326 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.562487    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-host\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.563326 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.562508    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-multus-cni-dir\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.563326 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.562515    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d-sys-fs\") pod \"aws-ebs-csi-driver-node-4n9vj\" (UID: \"f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n9vj"
Sep 17 20:49:22.563326 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.562545    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-host-run-k8s-cni-cncf-io\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.563326 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.562645    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5caabfba-6743-414e-8d96-caf028817a3b-cni-binary-copy\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.563326 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.563006    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/67a00ce8-eae6-48cb-896a-bb6de23904e9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rw72g\" (UID: \"67a00ce8-eae6-48cb-896a-bb6de23904e9\") " pod="openshift-multus/multus-additional-cni-plugins-rw72g"
Sep 17 20:49:22.563326 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.563044    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-sys\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.563326 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.563072    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67a00ce8-eae6-48cb-896a-bb6de23904e9-system-cni-dir\") pod \"multus-additional-cni-plugins-rw72g\" (UID: \"67a00ce8-eae6-48cb-896a-bb6de23904e9\") " pod="openshift-multus/multus-additional-cni-plugins-rw72g"
Sep 17 20:49:22.563326 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.563091    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-hostroot\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.563326 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.563142    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-host-run-netns\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.563326 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.563177    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-host-run-multus-certs\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.563326 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.563200    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d-socket-dir\") pod \"aws-ebs-csi-driver-node-4n9vj\" (UID: \"f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n9vj"
Sep 17 20:49:22.563326 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.563238    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67a00ce8-eae6-48cb-896a-bb6de23904e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rw72g\" (UID: \"67a00ce8-eae6-48cb-896a-bb6de23904e9\") " pod="openshift-multus/multus-additional-cni-plugins-rw72g"
Sep 17 20:49:22.563326 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.563312    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5caabfba-6743-414e-8d96-caf028817a3b-multus-daemon-config\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.563326 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.563320    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-multus-socket-dir-parent\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.564124 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.563321    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-hostroot\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.564124 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.563331    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-host-var-lib-kubelet\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.564124 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.563412    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d-etc-selinux\") pod \"aws-ebs-csi-driver-node-4n9vj\" (UID: \"f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n9vj"
Sep 17 20:49:22.564124 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.563482    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5e2bd987-6d8a-457b-ad67-f1e6597864e5-serviceca\") pod \"node-ca-j8n57\" (UID: \"5e2bd987-6d8a-457b-ad67-f1e6597864e5\") " pod="openshift-image-registry/node-ca-j8n57"
Sep 17 20:49:22.564124 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.563548    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-host-run-multus-certs\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.564124 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.563562    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-lib-modules\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.564124 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.563588    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-etc-sysconfig\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.564124 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.563606    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4n9vj\" (UID: \"f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n9vj"
Sep 17 20:49:22.564124 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.563778    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67a00ce8-eae6-48cb-896a-bb6de23904e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rw72g\" (UID: \"67a00ce8-eae6-48cb-896a-bb6de23904e9\") " pod="openshift-multus/multus-additional-cni-plugins-rw72g"
Sep 17 20:49:22.564124 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.563825    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-host-var-lib-cni-bin\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.564124 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.563867    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-host-var-lib-kubelet\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.564124 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.563989    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a-dbus\") pod \"global-pull-secret-syncer-ltk4w\" (UID: \"3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a\") " pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:22.564124 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.564014    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d-registration-dir\") pod \"aws-ebs-csi-driver-node-4n9vj\" (UID: \"f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n9vj"
Sep 17 20:49:22.564124 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.564093    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-etc-kubernetes\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.564124 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.564097    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-host\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.564807 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.564146    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d-sys-fs\") pod \"aws-ebs-csi-driver-node-4n9vj\" (UID: \"f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n9vj"
Sep 17 20:49:22.564807 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.564188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzrc6\" (UniqueName: \"kubernetes.io/projected/5e2bd987-6d8a-457b-ad67-f1e6597864e5-kube-api-access-pzrc6\") pod \"node-ca-j8n57\" (UID: \"5e2bd987-6d8a-457b-ad67-f1e6597864e5\") " pod="openshift-image-registry/node-ca-j8n57"
Sep 17 20:49:22.564807 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.564258    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67a00ce8-eae6-48cb-896a-bb6de23904e9-cnibin\") pod \"multus-additional-cni-plugins-rw72g\" (UID: \"67a00ce8-eae6-48cb-896a-bb6de23904e9\") " pod="openshift-multus/multus-additional-cni-plugins-rw72g"
Sep 17 20:49:22.564807 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.564286    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-host-var-lib-cni-multus\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.564807 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.564318    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j6sf\" (UniqueName: \"kubernetes.io/projected/3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b-kube-api-access-2j6sf\") pod \"network-check-target-9tkmw\" (UID: \"3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b\") " pod="openshift-network-diagnostics/network-check-target-9tkmw"
Sep 17 20:49:22.564807 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.564332    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-system-cni-dir\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.564807 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.564406    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-run\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.564807 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.564463    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67a00ce8-eae6-48cb-896a-bb6de23904e9-cni-binary-copy\") pod \"multus-additional-cni-plugins-rw72g\" (UID: \"67a00ce8-eae6-48cb-896a-bb6de23904e9\") " pod="openshift-multus/multus-additional-cni-plugins-rw72g"
Sep 17 20:49:22.564807 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.564544    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67a00ce8-eae6-48cb-896a-bb6de23904e9-cnibin\") pod \"multus-additional-cni-plugins-rw72g\" (UID: \"67a00ce8-eae6-48cb-896a-bb6de23904e9\") " pod="openshift-multus/multus-additional-cni-plugins-rw72g"
Sep 17 20:49:22.564807 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.564576    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5e2bd987-6d8a-457b-ad67-f1e6597864e5-serviceca\") pod \"node-ca-j8n57\" (UID: \"5e2bd987-6d8a-457b-ad67-f1e6597864e5\") " pod="openshift-image-registry/node-ca-j8n57"
Sep 17 20:49:22.564807 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.564631    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-host-var-lib-cni-multus\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.564807 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.564287    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5caabfba-6743-414e-8d96-caf028817a3b-host-run-k8s-cni-cncf-io\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.564807 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.564662    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e2bd987-6d8a-457b-ad67-f1e6597864e5-host\") pod \"node-ca-j8n57\" (UID: \"5e2bd987-6d8a-457b-ad67-f1e6597864e5\") " pod="openshift-image-registry/node-ca-j8n57"
Sep 17 20:49:22.564807 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.564722    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce8d6a4d-e013-4e57-a540-14ddc06d7745-var-lib-kubelet\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.564807 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:22.564760    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:22.565447 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:22.564862    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a-original-pull-secret podName:3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a nodeName:}" failed. No retries permitted until 2026-09-17 20:49:23.064811495 +0000 UTC m=+4.675544704 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a-original-pull-secret") pod "global-pull-secret-syncer-ltk4w" (UID: "3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:22.565447 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.564982    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/582bf447-0797-4411-bce3-89e1c2455f6d-konnectivity-ca\") pod \"konnectivity-agent-6mj5z\" (UID: \"582bf447-0797-4411-bce3-89e1c2455f6d\") " pod="kube-system/konnectivity-agent-6mj5z"
Sep 17 20:49:22.565447 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.565053    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ce8d6a4d-e013-4e57-a540-14ddc06d7745-etc-tuned\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.566662 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.566520    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/582bf447-0797-4411-bce3-89e1c2455f6d-agent-certs\") pod \"konnectivity-agent-6mj5z\" (UID: \"582bf447-0797-4411-bce3-89e1c2455f6d\") " pod="kube-system/konnectivity-agent-6mj5z"
Sep 17 20:49:22.567949 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.567926    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce8d6a4d-e013-4e57-a540-14ddc06d7745-tmp\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.584996 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.584949    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhb2k\" (UniqueName: \"kubernetes.io/projected/67a00ce8-eae6-48cb-896a-bb6de23904e9-kube-api-access-dhb2k\") pod \"multus-additional-cni-plugins-rw72g\" (UID: \"67a00ce8-eae6-48cb-896a-bb6de23904e9\") " pod="openshift-multus/multus-additional-cni-plugins-rw72g"
Sep 17 20:49:22.601702 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.601679    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzrc6\" (UniqueName: \"kubernetes.io/projected/5e2bd987-6d8a-457b-ad67-f1e6597864e5-kube-api-access-pzrc6\") pod \"node-ca-j8n57\" (UID: \"5e2bd987-6d8a-457b-ad67-f1e6597864e5\") " pod="openshift-image-registry/node-ca-j8n57"
Sep 17 20:49:22.601702 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:22.601702    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:22.601833 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:22.601729    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:22.601833 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:22.601739    2566 projected.go:196] Error preparing data for projected volume kube-api-access-2j6sf for pod openshift-network-diagnostics/network-check-target-9tkmw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:22.601833 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:22.601818    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b-kube-api-access-2j6sf podName:3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b nodeName:}" failed. No retries permitted until 2026-09-17 20:49:23.101804262 +0000 UTC m=+4.712537492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2j6sf" (UniqueName: "kubernetes.io/projected/3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b-kube-api-access-2j6sf") pod "network-check-target-9tkmw" (UID: "3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:22.601931 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.601904    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4l7m\" (UniqueName: \"kubernetes.io/projected/ce8d6a4d-e013-4e57-a540-14ddc06d7745-kube-api-access-s4l7m\") pod \"tuned-bzvzr\" (UID: \"ce8d6a4d-e013-4e57-a540-14ddc06d7745\") " pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.615465 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.615443    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfkpc\" (UniqueName: \"kubernetes.io/projected/f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d-kube-api-access-lfkpc\") pod \"aws-ebs-csi-driver-node-4n9vj\" (UID: \"f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n9vj"
Sep 17 20:49:22.615888 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.615870    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd24m\" (UniqueName: \"kubernetes.io/projected/5caabfba-6743-414e-8d96-caf028817a3b-kube-api-access-bd24m\") pod \"multus-h5f4t\" (UID: \"5caabfba-6743-414e-8d96-caf028817a3b\") " pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.615976 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.615928    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkq6j\" (UniqueName: \"kubernetes.io/projected/21362652-a68e-4d69-8cea-69002798058b-kube-api-access-tkq6j\") pod \"iptables-alerter-4nh74\" (UID: \"21362652-a68e-4d69-8cea-69002798058b\") " pod="openshift-network-operator/iptables-alerter-4nh74"
Sep 17 20:49:22.655296 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.655271    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j7t6x"
Sep 17 20:49:22.667854 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.667833    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:22.675658 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.675642    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6mj5z"
Sep 17 20:49:22.682150 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.682133    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j8n57"
Sep 17 20:49:22.688689 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.688668    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bzvzr"
Sep 17 20:49:22.695439 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.695424    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rw72g"
Sep 17 20:49:22.702700 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.702681    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n9vj"
Sep 17 20:49:22.711200 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.711183    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4nh74"
Sep 17 20:49:22.717771 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.717754    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h5f4t"
Sep 17 20:49:22.801396 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.801372    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:22.967183 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:22.967158    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-metrics-certs\") pod \"network-metrics-daemon-sknnf\" (UID: \"8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7\") " pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:49:22.967365 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:22.967343    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:22.967443 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:22.967432    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-metrics-certs podName:8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:23.967407738 +0000 UTC m=+5.578140969 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-metrics-certs") pod "network-metrics-daemon-sknnf" (UID: "8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:23.068157 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:23.068118    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a-original-pull-secret\") pod \"global-pull-secret-syncer-ltk4w\" (UID: \"3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a\") " pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:23.068334 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:23.068248    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:23.068334 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:23.068309    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a-original-pull-secret podName:3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a nodeName:}" failed. No retries permitted until 2026-09-17 20:49:24.068292084 +0000 UTC m=+5.679025297 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a-original-pull-secret") pod "global-pull-secret-syncer-ltk4w" (UID: "3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:23.168441 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:23.168413    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j6sf\" (UniqueName: \"kubernetes.io/projected/3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b-kube-api-access-2j6sf\") pod \"network-check-target-9tkmw\" (UID: \"3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b\") " pod="openshift-network-diagnostics/network-check-target-9tkmw"
Sep 17 20:49:23.168627 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:23.168609    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:23.168680 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:23.168632    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:23.168680 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:23.168646    2566 projected.go:196] Error preparing data for projected volume kube-api-access-2j6sf for pod openshift-network-diagnostics/network-check-target-9tkmw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:23.168764 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:23.168707    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b-kube-api-access-2j6sf podName:3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b nodeName:}" failed. No retries permitted until 2026-09-17 20:49:24.168689542 +0000 UTC m=+5.779422771 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2j6sf" (UniqueName: "kubernetes.io/projected/3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b-kube-api-access-2j6sf") pod "network-check-target-9tkmw" (UID: "3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:23.346879 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:23.346838    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f5d4daa_933c_47d6_b93a_6f83ed7ab23f.slice/crio-5be59d93d1c1bcb6bf90ece7a668c60b753ac44f6c0334f780cf9a0c19219edf WatchSource:0}: Error finding container 5be59d93d1c1bcb6bf90ece7a668c60b753ac44f6c0334f780cf9a0c19219edf: Status 404 returned error can't find the container with id 5be59d93d1c1bcb6bf90ece7a668c60b753ac44f6c0334f780cf9a0c19219edf
Sep 17 20:49:23.347739 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:23.347712    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce8d6a4d_e013_4e57_a540_14ddc06d7745.slice/crio-1f680898c63b23ac457069bcd1f4b9f9a800a20d2bd21cfd50544aaf687a23cf WatchSource:0}: Error finding container 1f680898c63b23ac457069bcd1f4b9f9a800a20d2bd21cfd50544aaf687a23cf: Status 404 returned error can't find the container with id 1f680898c63b23ac457069bcd1f4b9f9a800a20d2bd21cfd50544aaf687a23cf
Sep 17 20:49:23.349054 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:23.349012    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a00ce8_eae6_48cb_896a_bb6de23904e9.slice/crio-9eface32840fa9eacdf659d1bd496f216e2b6c0381411062d6937f9bfcd97d15 WatchSource:0}: Error finding container 9eface32840fa9eacdf659d1bd496f216e2b6c0381411062d6937f9bfcd97d15: Status 404 returned error can't find the container with id 9eface32840fa9eacdf659d1bd496f216e2b6c0381411062d6937f9bfcd97d15
Sep 17 20:49:23.350162 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:23.350138    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod402f55a4_24b6_47c6_b719_9c6a5b473434.slice/crio-a2cb5e7d306a686486b6d1a0a32789b3cad269a80e1cf24b7673c15f376c390b WatchSource:0}: Error finding container a2cb5e7d306a686486b6d1a0a32789b3cad269a80e1cf24b7673c15f376c390b: Status 404 returned error can't find the container with id a2cb5e7d306a686486b6d1a0a32789b3cad269a80e1cf24b7673c15f376c390b
Sep 17 20:49:23.351435 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:23.351413    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e2bd987_6d8a_457b_ad67_f1e6597864e5.slice/crio-5b9e48aa8f2f8502df0b748b39fbcd6888f29bd4ae06744c21050e29abe95c13 WatchSource:0}: Error finding container 5b9e48aa8f2f8502df0b748b39fbcd6888f29bd4ae06744c21050e29abe95c13: Status 404 returned error can't find the container with id 5b9e48aa8f2f8502df0b748b39fbcd6888f29bd4ae06744c21050e29abe95c13
Sep 17 20:49:23.354423 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:23.354392    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod582bf447_0797_4411_bce3_89e1c2455f6d.slice/crio-d28ef05b11ee3dd2b5f40e9075a3e555f86568698375bc6dce8f3762921f1949 WatchSource:0}: Error finding container d28ef05b11ee3dd2b5f40e9075a3e555f86568698375bc6dce8f3762921f1949: Status 404 returned error can't find the container with id d28ef05b11ee3dd2b5f40e9075a3e555f86568698375bc6dce8f3762921f1949
Sep 17 20:49:23.355878 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:23.355733    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3aafbcb_ba41_4703_8ab9_3f54aaf5e61d.slice/crio-d50b1275594db3f7b236fcc90fcc93b1e7124a546f3b5dd3323c3d01904b547e WatchSource:0}: Error finding container d50b1275594db3f7b236fcc90fcc93b1e7124a546f3b5dd3323c3d01904b547e: Status 404 returned error can't find the container with id d50b1275594db3f7b236fcc90fcc93b1e7124a546f3b5dd3323c3d01904b547e
Sep 17 20:49:23.356881 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:23.356600    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5caabfba_6743_414e_8d96_caf028817a3b.slice/crio-f79249db3de8d5ae39954de1b2bd38b8e2e88d56f93ccc0119646f60940a3c0b WatchSource:0}: Error finding container f79249db3de8d5ae39954de1b2bd38b8e2e88d56f93ccc0119646f60940a3c0b: Status 404 returned error can't find the container with id f79249db3de8d5ae39954de1b2bd38b8e2e88d56f93ccc0119646f60940a3c0b
Sep 17 20:49:23.357860 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:23.357760    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21362652_a68e_4d69_8cea_69002798058b.slice/crio-096ad737a463cb2dabab338dfd6fed7d874ea510db45428ad68e642d2bf47ad5 WatchSource:0}: Error finding container 096ad737a463cb2dabab338dfd6fed7d874ea510db45428ad68e642d2bf47ad5: Status 404 returned error can't find the container with id 096ad737a463cb2dabab338dfd6fed7d874ea510db45428ad68e642d2bf47ad5
Sep 17 20:49:23.454242 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:23.454055    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:49:23.454386 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:23.454343    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sknnf" podUID="8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7"
Sep 17 20:49:23.466023 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:23.465994    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-140.ec2.internal" event={"ID":"e97c66319ae520682604e45d82c8b847","Type":"ContainerStarted","Data":"5732758bb15d74f59a3573cdab00e6743099774ac3217f4d943d0fb9f509fe72"}
Sep 17 20:49:23.469829 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:23.469758    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h5f4t" event={"ID":"5caabfba-6743-414e-8d96-caf028817a3b","Type":"ContainerStarted","Data":"f79249db3de8d5ae39954de1b2bd38b8e2e88d56f93ccc0119646f60940a3c0b"}
Sep 17 20:49:23.471344 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:23.471319    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n9vj" event={"ID":"f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d","Type":"ContainerStarted","Data":"d50b1275594db3f7b236fcc90fcc93b1e7124a546f3b5dd3323c3d01904b547e"}
Sep 17 20:49:23.472583 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:23.472543    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6mj5z" event={"ID":"582bf447-0797-4411-bce3-89e1c2455f6d","Type":"ContainerStarted","Data":"d28ef05b11ee3dd2b5f40e9075a3e555f86568698375bc6dce8f3762921f1949"}
Sep 17 20:49:23.473810 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:23.473766    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j8n57" event={"ID":"5e2bd987-6d8a-457b-ad67-f1e6597864e5","Type":"ContainerStarted","Data":"5b9e48aa8f2f8502df0b748b39fbcd6888f29bd4ae06744c21050e29abe95c13"}
Sep 17 20:49:23.475093 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:23.475059    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bzvzr" event={"ID":"ce8d6a4d-e013-4e57-a540-14ddc06d7745","Type":"ContainerStarted","Data":"1f680898c63b23ac457069bcd1f4b9f9a800a20d2bd21cfd50544aaf687a23cf"}
Sep 17 20:49:23.476270 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:23.476247    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7" event={"ID":"402f55a4-24b6-47c6-b719-9c6a5b473434","Type":"ContainerStarted","Data":"a2cb5e7d306a686486b6d1a0a32789b3cad269a80e1cf24b7673c15f376c390b"}
Sep 17 20:49:23.477633 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:23.477614    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4nh74" event={"ID":"21362652-a68e-4d69-8cea-69002798058b","Type":"ContainerStarted","Data":"096ad737a463cb2dabab338dfd6fed7d874ea510db45428ad68e642d2bf47ad5"}
Sep 17 20:49:23.478800 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:23.478779    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rw72g" event={"ID":"67a00ce8-eae6-48cb-896a-bb6de23904e9","Type":"ContainerStarted","Data":"9eface32840fa9eacdf659d1bd496f216e2b6c0381411062d6937f9bfcd97d15"}
Sep 17 20:49:23.480166 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:23.480148    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j7t6x" event={"ID":"5f5d4daa-933c-47d6-b93a-6f83ed7ab23f","Type":"ContainerStarted","Data":"5be59d93d1c1bcb6bf90ece7a668c60b753ac44f6c0334f780cf9a0c19219edf"}
Sep 17 20:49:23.489928 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:23.489885    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-140.ec2.internal" podStartSLOduration=2.489870117 podStartE2EDuration="2.489870117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:23.489706952 +0000 UTC m=+5.100440181" watchObservedRunningTime="2026-09-17 20:49:23.489870117 +0000 UTC m=+5.100603348"
Sep 17 20:49:23.490268 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:23.490230    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 20:44:22 +0000 UTC" deadline="2028-05-16 07:20:32.123729646 +0000 UTC"
Sep 17 20:49:23.490268 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:23.490265    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14554h31m8.633466884s"
Sep 17 20:49:23.974860 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:23.974300    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-metrics-certs\") pod \"network-metrics-daemon-sknnf\" (UID: \"8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7\") " pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:49:23.974860 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:23.974472    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:23.974860 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:23.974535    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-metrics-certs podName:8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:25.974516121 +0000 UTC m=+7.585249337 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-metrics-certs") pod "network-metrics-daemon-sknnf" (UID: "8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:24.076059 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:24.075500    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a-original-pull-secret\") pod \"global-pull-secret-syncer-ltk4w\" (UID: \"3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a\") " pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:24.076059 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:24.075638    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:24.076059 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:24.075709    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a-original-pull-secret podName:3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a nodeName:}" failed. No retries permitted until 2026-09-17 20:49:26.0756914 +0000 UTC m=+7.686424615 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a-original-pull-secret") pod "global-pull-secret-syncer-ltk4w" (UID: "3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:24.176809 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:24.176773    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j6sf\" (UniqueName: \"kubernetes.io/projected/3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b-kube-api-access-2j6sf\") pod \"network-check-target-9tkmw\" (UID: \"3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b\") " pod="openshift-network-diagnostics/network-check-target-9tkmw"
Sep 17 20:49:24.176982 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:24.176943    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:24.176982 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:24.176962    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:24.176982 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:24.176974    2566 projected.go:196] Error preparing data for projected volume kube-api-access-2j6sf for pod openshift-network-diagnostics/network-check-target-9tkmw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:24.177122 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:24.177032    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b-kube-api-access-2j6sf podName:3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b nodeName:}" failed. No retries permitted until 2026-09-17 20:49:26.177014138 +0000 UTC m=+7.787747347 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2j6sf" (UniqueName: "kubernetes.io/projected/3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b-kube-api-access-2j6sf") pod "network-check-target-9tkmw" (UID: "3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:24.455942 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:24.455371    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tkmw"
Sep 17 20:49:24.455942 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:24.455493    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tkmw" podUID="3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b"
Sep 17 20:49:24.456665 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:24.456528    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:24.456665 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:24.456622    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltk4w" podUID="3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a"
Sep 17 20:49:24.490850 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:24.490817    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 20:44:22 +0000 UTC" deadline="2028-03-21 21:47:15.896932604 +0000 UTC"
Sep 17 20:49:24.490850 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:24.490850    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13224h57m51.406087097s"
Sep 17 20:49:24.493073 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:24.493048    2566 generic.go:356] "Generic (PLEG): container finished" podID="cce51aae58d840086c38254c622dda75" containerID="110262ebbd28f1aebc3492637bccde1f9ebcd24630b8a6e753b47e536b6ac487" exitCode=0
Sep 17 20:49:24.495289 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:24.493852    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal" event={"ID":"cce51aae58d840086c38254c622dda75","Type":"ContainerDied","Data":"110262ebbd28f1aebc3492637bccde1f9ebcd24630b8a6e753b47e536b6ac487"}
Sep 17 20:49:25.454013 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:25.453988    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:49:25.454195 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:25.454110    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sknnf" podUID="8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7"
Sep 17 20:49:25.505625 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:25.504971    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal" event={"ID":"cce51aae58d840086c38254c622dda75","Type":"ContainerStarted","Data":"1c5cce9b078c78d50f64a83dbb497b7563109ac172ac5e52a8dee6fd0958b64f"}
Sep 17 20:49:25.993276 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:25.991375    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-metrics-certs\") pod \"network-metrics-daemon-sknnf\" (UID: \"8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7\") " pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:49:25.993276 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:25.991536    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:25.993276 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:25.991627    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-metrics-certs podName:8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:29.99158577 +0000 UTC m=+11.602318997 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-metrics-certs") pod "network-metrics-daemon-sknnf" (UID: "8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:26.092075 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:26.092043    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a-original-pull-secret\") pod \"global-pull-secret-syncer-ltk4w\" (UID: \"3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a\") " pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:26.092256 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:26.092193    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:26.092329 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:26.092263    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a-original-pull-secret podName:3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a nodeName:}" failed. No retries permitted until 2026-09-17 20:49:30.092245056 +0000 UTC m=+11.702978268 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a-original-pull-secret") pod "global-pull-secret-syncer-ltk4w" (UID: "3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:26.192903 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:26.192871    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j6sf\" (UniqueName: \"kubernetes.io/projected/3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b-kube-api-access-2j6sf\") pod \"network-check-target-9tkmw\" (UID: \"3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b\") " pod="openshift-network-diagnostics/network-check-target-9tkmw"
Sep 17 20:49:26.193064 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:26.193053    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:26.193141 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:26.193075    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:26.193141 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:26.193086    2566 projected.go:196] Error preparing data for projected volume kube-api-access-2j6sf for pod openshift-network-diagnostics/network-check-target-9tkmw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:26.193141 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:26.193140    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b-kube-api-access-2j6sf podName:3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b nodeName:}" failed. No retries permitted until 2026-09-17 20:49:30.193123155 +0000 UTC m=+11.803856367 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2j6sf" (UniqueName: "kubernetes.io/projected/3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b-kube-api-access-2j6sf") pod "network-check-target-9tkmw" (UID: "3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:26.455421 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:26.455348    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tkmw"
Sep 17 20:49:26.455588 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:26.455480    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tkmw" podUID="3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b"
Sep 17 20:49:26.455886 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:26.455867    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:26.455986 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:26.455965    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltk4w" podUID="3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a"
Sep 17 20:49:27.454128 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:27.454095    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:49:27.454635 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:27.454248    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sknnf" podUID="8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7"
Sep 17 20:49:28.454398 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:28.454348    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tkmw"
Sep 17 20:49:28.454857 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:28.454498    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tkmw" podUID="3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b"
Sep 17 20:49:28.454857 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:28.454540    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:28.454857 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:28.454611    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltk4w" podUID="3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a"
Sep 17 20:49:29.454619 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:29.454588    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:49:29.455044 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:29.454752    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sknnf" podUID="8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7"
Sep 17 20:49:30.023793 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:30.023757    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-metrics-certs\") pod \"network-metrics-daemon-sknnf\" (UID: \"8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7\") " pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:49:30.023997 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:30.023901    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:30.023997 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:30.023972    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-metrics-certs podName:8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:38.023953089 +0000 UTC m=+19.634686297 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-metrics-certs") pod "network-metrics-daemon-sknnf" (UID: "8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:30.125322 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:30.124780    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a-original-pull-secret\") pod \"global-pull-secret-syncer-ltk4w\" (UID: \"3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a\") " pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:30.125322 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:30.124922    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:30.125322 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:30.124984    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a-original-pull-secret podName:3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a nodeName:}" failed. No retries permitted until 2026-09-17 20:49:38.124965667 +0000 UTC m=+19.735698880 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a-original-pull-secret") pod "global-pull-secret-syncer-ltk4w" (UID: "3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:30.226098 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:30.226062    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j6sf\" (UniqueName: \"kubernetes.io/projected/3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b-kube-api-access-2j6sf\") pod \"network-check-target-9tkmw\" (UID: \"3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b\") " pod="openshift-network-diagnostics/network-check-target-9tkmw"
Sep 17 20:49:30.226278 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:30.226183    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:30.226278 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:30.226197    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:30.226278 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:30.226220    2566 projected.go:196] Error preparing data for projected volume kube-api-access-2j6sf for pod openshift-network-diagnostics/network-check-target-9tkmw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:30.226278 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:30.226267    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b-kube-api-access-2j6sf podName:3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b nodeName:}" failed. No retries permitted until 2026-09-17 20:49:38.226255447 +0000 UTC m=+19.836988655 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2j6sf" (UniqueName: "kubernetes.io/projected/3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b-kube-api-access-2j6sf") pod "network-check-target-9tkmw" (UID: "3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:30.458478 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:30.458407    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:30.458788 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:30.458542    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltk4w" podUID="3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a"
Sep 17 20:49:30.458901 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:30.458878    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tkmw"
Sep 17 20:49:30.458989 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:30.458969    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tkmw" podUID="3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b"
Sep 17 20:49:31.455025 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:31.455000    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:49:31.455169 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:31.455138    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sknnf" podUID="8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7"
Sep 17 20:49:32.454085 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:32.454038    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tkmw"
Sep 17 20:49:32.454574 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:32.454038    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:32.454574 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:32.454202    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tkmw" podUID="3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b"
Sep 17 20:49:32.454574 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:32.454298    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltk4w" podUID="3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a"
Sep 17 20:49:33.454886 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:33.454850    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:49:33.455311 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:33.455014    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sknnf" podUID="8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7"
Sep 17 20:49:34.454452 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:34.454418    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:34.454731 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:34.454450    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tkmw"
Sep 17 20:49:34.454731 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:34.454543    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltk4w" podUID="3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a"
Sep 17 20:49:34.454731 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:34.454619    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tkmw" podUID="3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b"
Sep 17 20:49:35.454523 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:35.454487    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:49:35.454976 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:35.454639    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sknnf" podUID="8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7"
Sep 17 20:49:36.454504 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:36.454468    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tkmw"
Sep 17 20:49:36.454504 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:36.454504    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:36.455017 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:36.454587    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tkmw" podUID="3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b"
Sep 17 20:49:36.455017 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:36.454704    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltk4w" podUID="3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a"
Sep 17 20:49:37.454397 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:37.454367    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:49:37.454648 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:37.454479    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sknnf" podUID="8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7"
Sep 17 20:49:38.088034 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:38.088000    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-metrics-certs\") pod \"network-metrics-daemon-sknnf\" (UID: \"8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7\") " pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:49:38.088438 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:38.088125    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:38.088438 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:38.088186    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-metrics-certs podName:8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:54.088167621 +0000 UTC m=+35.698900835 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-metrics-certs") pod "network-metrics-daemon-sknnf" (UID: "8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:38.188505 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:38.188465    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a-original-pull-secret\") pod \"global-pull-secret-syncer-ltk4w\" (UID: \"3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a\") " pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:38.188704 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:38.188584    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:38.188704 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:38.188655    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a-original-pull-secret podName:3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a nodeName:}" failed. No retries permitted until 2026-09-17 20:49:54.188635901 +0000 UTC m=+35.799369112 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a-original-pull-secret") pod "global-pull-secret-syncer-ltk4w" (UID: "3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:38.288867 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:38.288836    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j6sf\" (UniqueName: \"kubernetes.io/projected/3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b-kube-api-access-2j6sf\") pod \"network-check-target-9tkmw\" (UID: \"3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b\") " pod="openshift-network-diagnostics/network-check-target-9tkmw"
Sep 17 20:49:38.289033 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:38.289016    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:38.289095 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:38.289040    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:38.289095 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:38.289053    2566 projected.go:196] Error preparing data for projected volume kube-api-access-2j6sf for pod openshift-network-diagnostics/network-check-target-9tkmw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:38.289192 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:38.289118    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b-kube-api-access-2j6sf podName:3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b nodeName:}" failed. No retries permitted until 2026-09-17 20:49:54.28909944 +0000 UTC m=+35.899832664 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2j6sf" (UniqueName: "kubernetes.io/projected/3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b-kube-api-access-2j6sf") pod "network-check-target-9tkmw" (UID: "3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:38.454314 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:38.454236    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tkmw"
Sep 17 20:49:38.454480 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:38.454391    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tkmw" podUID="3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b"
Sep 17 20:49:38.454480 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:38.454430    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:38.454576 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:38.454510    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltk4w" podUID="3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a"
Sep 17 20:49:39.454662 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:39.454633    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:49:39.455144 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:39.454743    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sknnf" podUID="8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7"
Sep 17 20:49:40.454869 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:40.454840    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tkmw"
Sep 17 20:49:40.455219 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:40.454929    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tkmw" podUID="3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b"
Sep 17 20:49:40.455219 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:40.454983    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:40.455219 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:40.455054    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltk4w" podUID="3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a"
Sep 17 20:49:41.454422 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:41.454106    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:49:41.454526 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:41.454508    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sknnf" podUID="8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7"
Sep 17 20:49:41.540543 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:41.539820    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h5f4t" event={"ID":"5caabfba-6743-414e-8d96-caf028817a3b","Type":"ContainerStarted","Data":"216d590ace5f980f7b9c3b5cd01ef85f55fea6f5d4f89408026934eb8ae460dc"}
Sep 17 20:49:41.543416 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:41.543388    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n9vj" event={"ID":"f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d","Type":"ContainerStarted","Data":"6024129243b1c85a9b420a42fdeeec69ede015a707c313b132987379739dc4dd"}
Sep 17 20:49:41.545132 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:41.545087    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6mj5z" event={"ID":"582bf447-0797-4411-bce3-89e1c2455f6d","Type":"ContainerStarted","Data":"b6481ff5eb695527a2eecf5b6b52dfcac9f8a9fa7430e9b1456d6298b521bfba"}
Sep 17 20:49:41.546824 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:41.546799    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j8n57" event={"ID":"5e2bd987-6d8a-457b-ad67-f1e6597864e5","Type":"ContainerStarted","Data":"f7546910b5177d22190e31dcd22afe26f754ac4c8609e9baa97f80f498111b39"}
Sep 17 20:49:41.548486 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:41.548448    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bzvzr" event={"ID":"ce8d6a4d-e013-4e57-a540-14ddc06d7745","Type":"ContainerStarted","Data":"bdcd1393f09d907f4f267dd57a55341a67228c5a4daf02be18ff0e9d51b1024f"}
Sep 17 20:49:41.551730 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:41.551709    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnqx7_402f55a4-24b6-47c6-b719-9c6a5b473434/ovn-acl-logging/0.log"
Sep 17 20:49:41.552109 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:41.552085    2566 generic.go:356] "Generic (PLEG): container finished" podID="402f55a4-24b6-47c6-b719-9c6a5b473434" containerID="5719f62322bd6d35a4b87360016a71c4c4bd702aad860c20cbf5b44c00a4380b" exitCode=1
Sep 17 20:49:41.552201 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:41.552142    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7" event={"ID":"402f55a4-24b6-47c6-b719-9c6a5b473434","Type":"ContainerStarted","Data":"94ad555190ea061eb8ae23f1b1e6954d1364a62926ce399655c6e06675a69617"}
Sep 17 20:49:41.552271 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:41.552222    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7" event={"ID":"402f55a4-24b6-47c6-b719-9c6a5b473434","Type":"ContainerStarted","Data":"4cb7ca053e120bf6f26f1414474564328a0691c1fc53289de29ae36ee176d470"}
Sep 17 20:49:41.552331 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:41.552270    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7" event={"ID":"402f55a4-24b6-47c6-b719-9c6a5b473434","Type":"ContainerStarted","Data":"ff1afd60d3f84069a3ffa093b3e4698f3c05cac14f6a7b98dd45e29385de6605"}
Sep 17 20:49:41.552331 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:41.552312    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7" event={"ID":"402f55a4-24b6-47c6-b719-9c6a5b473434","Type":"ContainerStarted","Data":"ca46935ce9240ee1b36e9998eba958a370022ca5da30bc5164bd4d49943e040a"}
Sep 17 20:49:41.552436 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:41.552349    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7" event={"ID":"402f55a4-24b6-47c6-b719-9c6a5b473434","Type":"ContainerDied","Data":"5719f62322bd6d35a4b87360016a71c4c4bd702aad860c20cbf5b44c00a4380b"}
Sep 17 20:49:41.552436 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:41.552398    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7" event={"ID":"402f55a4-24b6-47c6-b719-9c6a5b473434","Type":"ContainerStarted","Data":"c75abadffe1b6d4f6350dfefa44868012edd9b3d38367178e4a79fe1e1f4e73d"}
Sep 17 20:49:41.554084 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:41.554061    2566 generic.go:356] "Generic (PLEG): container finished" podID="67a00ce8-eae6-48cb-896a-bb6de23904e9" containerID="2d0ee605d310b65aa37db3d93f51d08a6578b3ce893f81713efafb90c2e080f9" exitCode=0
Sep 17 20:49:41.554173 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:41.554091    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rw72g" event={"ID":"67a00ce8-eae6-48cb-896a-bb6de23904e9","Type":"ContainerDied","Data":"2d0ee605d310b65aa37db3d93f51d08a6578b3ce893f81713efafb90c2e080f9"}
Sep 17 20:49:41.555952 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:41.555932    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j7t6x" event={"ID":"5f5d4daa-933c-47d6-b93a-6f83ed7ab23f","Type":"ContainerStarted","Data":"47390bb23e67bddb4cb0bb9750943ef210e160afc2b00b0570b25555a8df907c"}
Sep 17 20:49:41.558090 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:41.558052    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-h5f4t" podStartSLOduration=4.209185357 podStartE2EDuration="21.558042064s" totalImagesPullingTime="17.348856707s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:41.55800828 +0000 UTC m=+23.168741528" watchObservedRunningTime="2026-09-17 20:49:41.558042064 +0000 UTC m=+23.168775293"
Sep 17 20:49:41.558353 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:41.558309    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal" podStartSLOduration=20.558293283 podStartE2EDuration="20.558293283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:25.522345215 +0000 UTC m=+7.133078444" watchObservedRunningTime="2026-09-17 20:49:41.558293283 +0000 UTC m=+23.169026514"
Sep 17 20:49:41.575077 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:41.575044    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bzvzr" podStartSLOduration=4.239060548 podStartE2EDuration="21.57503138s" totalImagesPullingTime="17.335970832s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:41.574551365 +0000 UTC m=+23.185284622" watchObservedRunningTime="2026-09-17 20:49:41.57503138 +0000 UTC m=+23.185764619"
Sep 17 20:49:41.590306 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:41.590247    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-j7t6x" podStartSLOduration=4.255411874 podStartE2EDuration="21.590235088s" totalImagesPullingTime="17.334823214s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:41.589623056 +0000 UTC m=+23.200356287" watchObservedRunningTime="2026-09-17 20:49:41.590235088 +0000 UTC m=+23.200968314"
Sep 17 20:49:41.621361 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:41.621315    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-j8n57" podStartSLOduration=4.293524252 podStartE2EDuration="21.621301809s" totalImagesPullingTime="17.327777557s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:41.621287382 +0000 UTC m=+23.232020617" watchObservedRunningTime="2026-09-17 20:49:41.621301809 +0000 UTC m=+23.232035017"
Sep 17 20:49:41.621872 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:41.621840    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6mj5z" podStartSLOduration=11.642664966 podStartE2EDuration="21.621833264s" totalImagesPullingTime="9.979168298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:41.606997666 +0000 UTC m=+23.217730896" watchObservedRunningTime="2026-09-17 20:49:41.621833264 +0000 UTC m=+23.232566494"
Sep 17 20:49:42.028133 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:42.028113    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 20:49:42.438828 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:42.438735    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T20:49:42.028130413Z","UUID":"fc7bba6c-fc17-42af-8931-fc2359cc583a","Handler":null,"Name":"","Endpoint":""}
Sep 17 20:49:42.440416 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:42.440381    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 20:49:42.440416 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:42.440418    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 20:49:42.454490 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:42.454463    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tkmw"
Sep 17 20:49:42.454606 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:42.454574    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tkmw" podUID="3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b"
Sep 17 20:49:42.454677 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:42.454636    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:42.454726 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:42.454708    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltk4w" podUID="3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a"
Sep 17 20:49:42.560760 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:42.560726    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n9vj" event={"ID":"f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d","Type":"ContainerStarted","Data":"d6e906390be808d63f1897246e47e92a0b371e8cd504a487e9b394e0b6f85063"}
Sep 17 20:49:42.562838 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:42.562469    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4nh74" event={"ID":"21362652-a68e-4d69-8cea-69002798058b","Type":"ContainerStarted","Data":"09e78e51e5eb6fee3d00c1a7df75d1b9b3ec01b113ed85d6a91191de49231e3a"}
Sep 17 20:49:42.587837 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:42.587795    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4nh74" podStartSLOduration=5.263879058 podStartE2EDuration="22.587779822s" totalImagesPullingTime="17.323900764s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:42.587405043 +0000 UTC m=+24.198138275" watchObservedRunningTime="2026-09-17 20:49:42.587779822 +0000 UTC m=+24.198513052"
Sep 17 20:49:43.454253 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:43.454225    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:49:43.454394 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:43.454336    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sknnf" podUID="8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7"
Sep 17 20:49:43.566550 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:43.566519    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n9vj" event={"ID":"f3aafbcb-ba41-4703-8ab9-3f54aaf5e61d","Type":"ContainerStarted","Data":"9cdd1b5fdab2374e1e03aa6ec8aba913c0c509feeff3115ebde8bcb2345a34fe"}
Sep 17 20:49:43.590180 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:43.590134    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n9vj" podStartSLOduration=4.079006217 podStartE2EDuration="23.590113825s" totalImagesPullingTime="19.511107608s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:43.587617487 +0000 UTC m=+25.198350736" watchObservedRunningTime="2026-09-17 20:49:43.590113825 +0000 UTC m=+25.200847054"
Sep 17 20:49:44.080093 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:44.079846    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6mj5z"
Sep 17 20:49:44.083222 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:44.081009    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6mj5z"
Sep 17 20:49:44.454536 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:44.454462    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tkmw"
Sep 17 20:49:44.454710 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:44.454466    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:44.454710 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:44.454599    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tkmw" podUID="3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b"
Sep 17 20:49:44.454710 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:44.454692    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltk4w" podUID="3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a"
Sep 17 20:49:44.572453 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:44.572422    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnqx7_402f55a4-24b6-47c6-b719-9c6a5b473434/ovn-acl-logging/0.log"
Sep 17 20:49:44.572938 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:44.572912    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7" event={"ID":"402f55a4-24b6-47c6-b719-9c6a5b473434","Type":"ContainerStarted","Data":"30e4f8c1785d3ffa6cce1bb0a4326f293a23b6cbff362416900b8618c7146a31"}
Sep 17 20:49:45.454786 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:45.454760    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:49:45.454947 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:45.454873    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sknnf" podUID="8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7"
Sep 17 20:49:46.454560 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:46.454284    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tkmw"
Sep 17 20:49:46.455318 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:46.454355    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:46.455318 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:46.454664    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tkmw" podUID="3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b"
Sep 17 20:49:46.455318 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:46.454729    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltk4w" podUID="3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a"
Sep 17 20:49:46.580680 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:46.580655    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnqx7_402f55a4-24b6-47c6-b719-9c6a5b473434/ovn-acl-logging/0.log"
Sep 17 20:49:46.581041 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:46.581022    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7" event={"ID":"402f55a4-24b6-47c6-b719-9c6a5b473434","Type":"ContainerStarted","Data":"5d46ba18361fe8a158327b866d0b042500534ca990a859186b5e143ddf76a517"}
Sep 17 20:49:46.581282 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:46.581255    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:46.581426 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:46.581381    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:46.581484 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:46.581430    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:46.581517 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:46.581499    2566 scope.go:117] "RemoveContainer" containerID="5719f62322bd6d35a4b87360016a71c4c4bd702aad860c20cbf5b44c00a4380b"
Sep 17 20:49:46.583061 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:46.583037    2566 generic.go:356] "Generic (PLEG): container finished" podID="67a00ce8-eae6-48cb-896a-bb6de23904e9" containerID="52c6197d546a764877d030abc5eaa141107395339be71e4a32936b4671567558" exitCode=0
Sep 17 20:49:46.583140 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:46.583071    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rw72g" event={"ID":"67a00ce8-eae6-48cb-896a-bb6de23904e9","Type":"ContainerDied","Data":"52c6197d546a764877d030abc5eaa141107395339be71e4a32936b4671567558"}
Sep 17 20:49:46.598385 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:46.598363    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:46.598475 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:46.598465    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:49:47.454182 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:47.454144    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:49:47.454392 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:47.454378    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sknnf" podUID="8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7"
Sep 17 20:49:47.589855 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:47.589829    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnqx7_402f55a4-24b6-47c6-b719-9c6a5b473434/ovn-acl-logging/0.log"
Sep 17 20:49:47.590265 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:47.590235    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7" event={"ID":"402f55a4-24b6-47c6-b719-9c6a5b473434","Type":"ContainerStarted","Data":"d06ed6657ff36ac0634e60f3242b7fe3c218d68f88befdb3c297f569f1128036"}
Sep 17 20:49:47.621012 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:47.620970    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7" podStartSLOduration=10.231536754 podStartE2EDuration="27.620955599s" totalImagesPullingTime="17.389418845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:47.620291511 +0000 UTC m=+29.231024739" watchObservedRunningTime="2026-09-17 20:49:47.620955599 +0000 UTC m=+29.231688828"
Sep 17 20:49:47.762187 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:47.762156    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9tkmw"]
Sep 17 20:49:47.762347 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:47.762295    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tkmw"
Sep 17 20:49:47.762407 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:47.762374    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tkmw" podUID="3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b"
Sep 17 20:49:47.764670 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:47.764647    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ltk4w"]
Sep 17 20:49:47.764792 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:47.764743    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:47.764891 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:47.764836    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltk4w" podUID="3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a"
Sep 17 20:49:47.765175 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:47.765153    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sknnf"]
Sep 17 20:49:47.765332 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:47.765277    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:49:47.765397 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:47.765367    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sknnf" podUID="8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7"
Sep 17 20:49:48.594167 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:48.594135    2566 generic.go:356] "Generic (PLEG): container finished" podID="67a00ce8-eae6-48cb-896a-bb6de23904e9" containerID="21f6f35b28b4dd187e73de41e409065ea594f369be8fd4f5e763b5503ecd7168" exitCode=0
Sep 17 20:49:48.594641 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:48.594188    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rw72g" event={"ID":"67a00ce8-eae6-48cb-896a-bb6de23904e9","Type":"ContainerDied","Data":"21f6f35b28b4dd187e73de41e409065ea594f369be8fd4f5e763b5503ecd7168"}
Sep 17 20:49:48.610648 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:48.610479    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6mj5z"
Sep 17 20:49:48.610773 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:48.610752    2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 20:49:48.611020 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:48.611002    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6mj5z"
Sep 17 20:49:49.454833 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:49.454797    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tkmw"
Sep 17 20:49:49.454833 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:49.454839    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:49.455076 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:49.454797    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:49:49.455076 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:49.454916    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tkmw" podUID="3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b"
Sep 17 20:49:49.455076 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:49.454983    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sknnf" podUID="8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7"
Sep 17 20:49:49.455076 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:49.455035    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltk4w" podUID="3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a"
Sep 17 20:49:50.601532 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:50.601501    2566 generic.go:356] "Generic (PLEG): container finished" podID="67a00ce8-eae6-48cb-896a-bb6de23904e9" containerID="0d54008b97943bdfcad717d808b6531baea90e9657c527f14eb3cd3b1b6ab280" exitCode=0
Sep 17 20:49:50.602065 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:50.601543    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rw72g" event={"ID":"67a00ce8-eae6-48cb-896a-bb6de23904e9","Type":"ContainerDied","Data":"0d54008b97943bdfcad717d808b6531baea90e9657c527f14eb3cd3b1b6ab280"}
Sep 17 20:49:51.454771 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:51.454300    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:51.454771 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:51.454318    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tkmw"
Sep 17 20:49:51.454771 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:51.454318    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:49:51.454771 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:51.454430    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltk4w" podUID="3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a"
Sep 17 20:49:51.454771 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:51.454500    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tkmw" podUID="3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b"
Sep 17 20:49:51.454771 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:51.454580    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sknnf" podUID="8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7"
Sep 17 20:49:53.397600 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.397567    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeReady"
Sep 17 20:49:53.398070 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.397704    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 20:49:53.450243 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.449929    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-78ncn"]
Sep 17 20:49:53.467396 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.467370    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vnd5t"]
Sep 17 20:49:53.467551 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.467444    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:53.467551 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.467454    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tkmw"
Sep 17 20:49:53.467551 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.467466    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-78ncn"
Sep 17 20:49:53.469108 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.468981    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:49:53.471689 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.470946    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 20:49:53.471689 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.471245    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-w6qzl\""
Sep 17 20:49:53.471689 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.471462    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 20:49:53.471689 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.471585    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 20:49:53.471689 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.471635    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 20:49:53.472012 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.471994    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 20:49:53.473125 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.473104    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lzdg7\""
Sep 17 20:49:53.473624 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.473355    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 20:49:53.473624 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.473580    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6gfd5\""
Sep 17 20:49:53.483427 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.483406    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vnd5t"]
Sep 17 20:49:53.483539 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.483450    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-78ncn"]
Sep 17 20:49:53.483539 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.483530    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vnd5t"
Sep 17 20:49:53.488331 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.488312    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 20:49:53.488421 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.488312    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 20:49:53.488421 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.488352    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 20:49:53.488525 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.488425    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7zl29\""
Sep 17 20:49:53.600168 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.600098    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-metrics-tls\") pod \"dns-default-78ncn\" (UID: \"ae0c2874-4c74-491b-8fa4-4ff0fcc84c00\") " pod="openshift-dns/dns-default-78ncn"
Sep 17 20:49:53.600168 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.600145    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2j6m\" (UniqueName: \"kubernetes.io/projected/850f96d3-cdee-4f2e-8e36-f32466aa1bbb-kube-api-access-t2j6m\") pod \"ingress-canary-vnd5t\" (UID: \"850f96d3-cdee-4f2e-8e36-f32466aa1bbb\") " pod="openshift-ingress-canary/ingress-canary-vnd5t"
Sep 17 20:49:53.600380 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.600169    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-config-volume\") pod \"dns-default-78ncn\" (UID: \"ae0c2874-4c74-491b-8fa4-4ff0fcc84c00\") " pod="openshift-dns/dns-default-78ncn"
Sep 17 20:49:53.600380 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.600255    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/850f96d3-cdee-4f2e-8e36-f32466aa1bbb-cert\") pod \"ingress-canary-vnd5t\" (UID: \"850f96d3-cdee-4f2e-8e36-f32466aa1bbb\") " pod="openshift-ingress-canary/ingress-canary-vnd5t"
Sep 17 20:49:53.600380 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.600319    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrzpf\" (UniqueName: \"kubernetes.io/projected/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-kube-api-access-rrzpf\") pod \"dns-default-78ncn\" (UID: \"ae0c2874-4c74-491b-8fa4-4ff0fcc84c00\") " pod="openshift-dns/dns-default-78ncn"
Sep 17 20:49:53.600507 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.600380    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-tmp-dir\") pod \"dns-default-78ncn\" (UID: \"ae0c2874-4c74-491b-8fa4-4ff0fcc84c00\") " pod="openshift-dns/dns-default-78ncn"
Sep 17 20:49:53.701284 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.701250    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2j6m\" (UniqueName: \"kubernetes.io/projected/850f96d3-cdee-4f2e-8e36-f32466aa1bbb-kube-api-access-t2j6m\") pod \"ingress-canary-vnd5t\" (UID: \"850f96d3-cdee-4f2e-8e36-f32466aa1bbb\") " pod="openshift-ingress-canary/ingress-canary-vnd5t"
Sep 17 20:49:53.701442 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.701305    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-config-volume\") pod \"dns-default-78ncn\" (UID: \"ae0c2874-4c74-491b-8fa4-4ff0fcc84c00\") " pod="openshift-dns/dns-default-78ncn"
Sep 17 20:49:53.701442 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.701338    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/850f96d3-cdee-4f2e-8e36-f32466aa1bbb-cert\") pod \"ingress-canary-vnd5t\" (UID: \"850f96d3-cdee-4f2e-8e36-f32466aa1bbb\") " pod="openshift-ingress-canary/ingress-canary-vnd5t"
Sep 17 20:49:53.701442 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.701361    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrzpf\" (UniqueName: \"kubernetes.io/projected/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-kube-api-access-rrzpf\") pod \"dns-default-78ncn\" (UID: \"ae0c2874-4c74-491b-8fa4-4ff0fcc84c00\") " pod="openshift-dns/dns-default-78ncn"
Sep 17 20:49:53.701442 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.701385    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-tmp-dir\") pod \"dns-default-78ncn\" (UID: \"ae0c2874-4c74-491b-8fa4-4ff0fcc84c00\") " pod="openshift-dns/dns-default-78ncn"
Sep 17 20:49:53.701442 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.701410    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-metrics-tls\") pod \"dns-default-78ncn\" (UID: \"ae0c2874-4c74-491b-8fa4-4ff0fcc84c00\") " pod="openshift-dns/dns-default-78ncn"
Sep 17 20:49:53.701720 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:53.701524    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:49:53.701720 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:53.701591    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-metrics-tls podName:ae0c2874-4c74-491b-8fa4-4ff0fcc84c00 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:54.201572253 +0000 UTC m=+35.812305475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-metrics-tls") pod "dns-default-78ncn" (UID: "ae0c2874-4c74-491b-8fa4-4ff0fcc84c00") : secret "dns-default-metrics-tls" not found
Sep 17 20:49:53.702239 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.702195    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-tmp-dir\") pod \"dns-default-78ncn\" (UID: \"ae0c2874-4c74-491b-8fa4-4ff0fcc84c00\") " pod="openshift-dns/dns-default-78ncn"
Sep 17 20:49:53.702239 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.702231    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-config-volume\") pod \"dns-default-78ncn\" (UID: \"ae0c2874-4c74-491b-8fa4-4ff0fcc84c00\") " pod="openshift-dns/dns-default-78ncn"
Sep 17 20:49:53.706063 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.706045    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/850f96d3-cdee-4f2e-8e36-f32466aa1bbb-cert\") pod \"ingress-canary-vnd5t\" (UID: \"850f96d3-cdee-4f2e-8e36-f32466aa1bbb\") " pod="openshift-ingress-canary/ingress-canary-vnd5t"
Sep 17 20:49:53.720597 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.720578    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2j6m\" (UniqueName: \"kubernetes.io/projected/850f96d3-cdee-4f2e-8e36-f32466aa1bbb-kube-api-access-t2j6m\") pod \"ingress-canary-vnd5t\" (UID: \"850f96d3-cdee-4f2e-8e36-f32466aa1bbb\") " pod="openshift-ingress-canary/ingress-canary-vnd5t"
Sep 17 20:49:53.721158 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.721140    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrzpf\" (UniqueName: \"kubernetes.io/projected/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-kube-api-access-rrzpf\") pod \"dns-default-78ncn\" (UID: \"ae0c2874-4c74-491b-8fa4-4ff0fcc84c00\") " pod="openshift-dns/dns-default-78ncn"
Sep 17 20:49:53.809562 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:53.809532    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vnd5t"
Sep 17 20:49:54.103444 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:54.103393    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-metrics-certs\") pod \"network-metrics-daemon-sknnf\" (UID: \"8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7\") " pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:49:54.103626 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:54.103557    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 20:49:54.103689 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:54.103634    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-metrics-certs podName:8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:26.103612948 +0000 UTC m=+67.714346156 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-metrics-certs") pod "network-metrics-daemon-sknnf" (UID: "8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7") : secret "metrics-daemon-secret" not found
Sep 17 20:49:54.204464 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:54.204425    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a-original-pull-secret\") pod \"global-pull-secret-syncer-ltk4w\" (UID: \"3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a\") " pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:54.204464 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:54.204485    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-metrics-tls\") pod \"dns-default-78ncn\" (UID: \"ae0c2874-4c74-491b-8fa4-4ff0fcc84c00\") " pod="openshift-dns/dns-default-78ncn"
Sep 17 20:49:54.204676 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:54.204629    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:49:54.204716 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:54.204682    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-metrics-tls podName:ae0c2874-4c74-491b-8fa4-4ff0fcc84c00 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:55.204669084 +0000 UTC m=+36.815402292 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-metrics-tls") pod "dns-default-78ncn" (UID: "ae0c2874-4c74-491b-8fa4-4ff0fcc84c00") : secret "dns-default-metrics-tls" not found
Sep 17 20:49:54.207071 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:54.207045    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a-original-pull-secret\") pod \"global-pull-secret-syncer-ltk4w\" (UID: \"3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a\") " pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:54.305292 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:54.305256    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j6sf\" (UniqueName: \"kubernetes.io/projected/3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b-kube-api-access-2j6sf\") pod \"network-check-target-9tkmw\" (UID: \"3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b\") " pod="openshift-network-diagnostics/network-check-target-9tkmw"
Sep 17 20:49:54.307949 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:54.307922    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j6sf\" (UniqueName: \"kubernetes.io/projected/3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b-kube-api-access-2j6sf\") pod \"network-check-target-9tkmw\" (UID: \"3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b\") " pod="openshift-network-diagnostics/network-check-target-9tkmw"
Sep 17 20:49:54.382824 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:54.382760    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tkmw"
Sep 17 20:49:54.404175 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:54.404145    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltk4w"
Sep 17 20:49:54.984055 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:54.983974    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ltk4w"]
Sep 17 20:49:54.984756 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:54.984733    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vnd5t"]
Sep 17 20:49:54.985589 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:54.985493    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9tkmw"]
Sep 17 20:49:55.002677 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:55.002654    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3825a8f7_bf7a_4e42_8fd3_80ec1bfc615b.slice/crio-a237eebae582b0696dc8aed164f5550a2ccbb46064650b3b02810d3066761dbf WatchSource:0}: Error finding container a237eebae582b0696dc8aed164f5550a2ccbb46064650b3b02810d3066761dbf: Status 404 returned error can't find the container with id a237eebae582b0696dc8aed164f5550a2ccbb46064650b3b02810d3066761dbf
Sep 17 20:49:55.013412 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:55.013383    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod850f96d3_cdee_4f2e_8e36_f32466aa1bbb.slice/crio-d13d646ee7972034f5926d349e5c05dbb39dab5ae4d330413409f4ac26d117e8 WatchSource:0}: Error finding container d13d646ee7972034f5926d349e5c05dbb39dab5ae4d330413409f4ac26d117e8: Status 404 returned error can't find the container with id d13d646ee7972034f5926d349e5c05dbb39dab5ae4d330413409f4ac26d117e8
Sep 17 20:49:55.013895 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:49:55.013877    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa9cc3a_9ccd_4894_b3ad_14ad007ced8a.slice/crio-6d7fdf7022e6699cb87e29633cfe1df5a6881df443f68391715b6885b17ca093 WatchSource:0}: Error finding container 6d7fdf7022e6699cb87e29633cfe1df5a6881df443f68391715b6885b17ca093: Status 404 returned error can't find the container with id 6d7fdf7022e6699cb87e29633cfe1df5a6881df443f68391715b6885b17ca093
Sep 17 20:49:55.213600 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:55.213571    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-metrics-tls\") pod \"dns-default-78ncn\" (UID: \"ae0c2874-4c74-491b-8fa4-4ff0fcc84c00\") " pod="openshift-dns/dns-default-78ncn"
Sep 17 20:49:55.213771 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:55.213712    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:49:55.213839 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:55.213774    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-metrics-tls podName:ae0c2874-4c74-491b-8fa4-4ff0fcc84c00 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:57.213755575 +0000 UTC m=+38.824488787 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-metrics-tls") pod "dns-default-78ncn" (UID: "ae0c2874-4c74-491b-8fa4-4ff0fcc84c00") : secret "dns-default-metrics-tls" not found
Sep 17 20:49:55.617968 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:55.617895    2566 generic.go:356] "Generic (PLEG): container finished" podID="67a00ce8-eae6-48cb-896a-bb6de23904e9" containerID="ab2fe6b9118a616a9677ca932e03b924d4109d9850a2c7f7a5942a7651e4dc5d" exitCode=0
Sep 17 20:49:55.618417 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:55.617979    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rw72g" event={"ID":"67a00ce8-eae6-48cb-896a-bb6de23904e9","Type":"ContainerDied","Data":"ab2fe6b9118a616a9677ca932e03b924d4109d9850a2c7f7a5942a7651e4dc5d"}
Sep 17 20:49:55.619789 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:55.619670    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vnd5t" event={"ID":"850f96d3-cdee-4f2e-8e36-f32466aa1bbb","Type":"ContainerStarted","Data":"d13d646ee7972034f5926d349e5c05dbb39dab5ae4d330413409f4ac26d117e8"}
Sep 17 20:49:55.621242 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:55.621203    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9tkmw" event={"ID":"3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b","Type":"ContainerStarted","Data":"a237eebae582b0696dc8aed164f5550a2ccbb46064650b3b02810d3066761dbf"}
Sep 17 20:49:55.622980 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:55.622954    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ltk4w" event={"ID":"3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a","Type":"ContainerStarted","Data":"6d7fdf7022e6699cb87e29633cfe1df5a6881df443f68391715b6885b17ca093"}
Sep 17 20:49:56.631188 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:56.631150    2566 generic.go:356] "Generic (PLEG): container finished" podID="67a00ce8-eae6-48cb-896a-bb6de23904e9" containerID="42706d921cc79edb22c62bfc9ed4e501e63b0790040ca39bd6939f4e7c9b7825" exitCode=0
Sep 17 20:49:56.631634 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:56.631204    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rw72g" event={"ID":"67a00ce8-eae6-48cb-896a-bb6de23904e9","Type":"ContainerDied","Data":"42706d921cc79edb22c62bfc9ed4e501e63b0790040ca39bd6939f4e7c9b7825"}
Sep 17 20:49:57.230035 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:57.229828    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-metrics-tls\") pod \"dns-default-78ncn\" (UID: \"ae0c2874-4c74-491b-8fa4-4ff0fcc84c00\") " pod="openshift-dns/dns-default-78ncn"
Sep 17 20:49:57.230288 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:57.229975    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:49:57.230288 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:49:57.230166    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-metrics-tls podName:ae0c2874-4c74-491b-8fa4-4ff0fcc84c00 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:01.230133611 +0000 UTC m=+42.840866824 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-metrics-tls") pod "dns-default-78ncn" (UID: "ae0c2874-4c74-491b-8fa4-4ff0fcc84c00") : secret "dns-default-metrics-tls" not found
Sep 17 20:49:57.639605 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:57.639531    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rw72g" event={"ID":"67a00ce8-eae6-48cb-896a-bb6de23904e9","Type":"ContainerStarted","Data":"216884cf20626d18cfe14c4839a447611530471e139bbf8f20bc2fa65f8bb099"}
Sep 17 20:49:57.670122 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:49:57.670063    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rw72g" podStartSLOduration=9.28540287 podStartE2EDuration="37.670049631s" totalImagesPullingTime="28.384646761s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:57.669285081 +0000 UTC m=+39.280018312" watchObservedRunningTime="2026-09-17 20:49:57.670049631 +0000 UTC m=+39.280782860"
Sep 17 20:50:01.258245 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:50:01.258185    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-metrics-tls\") pod \"dns-default-78ncn\" (UID: \"ae0c2874-4c74-491b-8fa4-4ff0fcc84c00\") " pod="openshift-dns/dns-default-78ncn"
Sep 17 20:50:01.258687 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:50:01.258337    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:50:01.258687 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:50:01.258428    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-metrics-tls podName:ae0c2874-4c74-491b-8fa4-4ff0fcc84c00 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:09.258411547 +0000 UTC m=+50.869144754 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-metrics-tls") pod "dns-default-78ncn" (UID: "ae0c2874-4c74-491b-8fa4-4ff0fcc84c00") : secret "dns-default-metrics-tls" not found
Sep 17 20:50:01.651694 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:50:01.651616    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vnd5t" event={"ID":"850f96d3-cdee-4f2e-8e36-f32466aa1bbb","Type":"ContainerStarted","Data":"033fe7826f22c8e30e18aac028bc13dba147cfa17199cb148bebfe34d49828ff"}
Sep 17 20:50:01.653086 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:50:01.653064    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9tkmw" event={"ID":"3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b","Type":"ContainerStarted","Data":"e4afc04608d37202390069e886f4568f15414ec678971e7e172ea84919e6dfb9"}
Sep 17 20:50:01.653191 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:50:01.653176    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9tkmw"
Sep 17 20:50:01.654518 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:50:01.654499    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ltk4w" event={"ID":"3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a","Type":"ContainerStarted","Data":"bc90e44c6268290147b0610cd07de4919e0f231a642f6b4e10fae96a48b99e34"}
Sep 17 20:50:01.671111 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:50:01.671072    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vnd5t" podStartSLOduration=3.043357045 podStartE2EDuration="8.671059664s" totalImagesPullingTime="5.627702619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:01.670486206 +0000 UTC m=+43.281219437" watchObservedRunningTime="2026-09-17 20:50:01.671059664 +0000 UTC m=+43.281792893"
Sep 17 20:50:01.694156 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:50:01.694114    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ltk4w" podStartSLOduration=36.054513452 podStartE2EDuration="41.694100194s" totalImagesPullingTime="5.639586742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:01.692954912 +0000 UTC m=+43.303688140" watchObservedRunningTime="2026-09-17 20:50:01.694100194 +0000 UTC m=+43.304833417"
Sep 17 20:50:01.713561 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:50:01.713515    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9tkmw" podStartSLOduration=36.051044854 podStartE2EDuration="41.713503721s" totalImagesPullingTime="5.662458867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:01.712823851 +0000 UTC m=+43.323557079" watchObservedRunningTime="2026-09-17 20:50:01.713503721 +0000 UTC m=+43.324236950"
Sep 17 20:50:09.306562 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:50:09.306521    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-metrics-tls\") pod \"dns-default-78ncn\" (UID: \"ae0c2874-4c74-491b-8fa4-4ff0fcc84c00\") " pod="openshift-dns/dns-default-78ncn"
Sep 17 20:50:09.306985 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:50:09.306645    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:50:09.306985 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:50:09.306716    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-metrics-tls podName:ae0c2874-4c74-491b-8fa4-4ff0fcc84c00 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:25.306698133 +0000 UTC m=+66.917431344 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-metrics-tls") pod "dns-default-78ncn" (UID: "ae0c2874-4c74-491b-8fa4-4ff0fcc84c00") : secret "dns-default-metrics-tls" not found
Sep 17 20:50:18.603990 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:50:18.603963    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bnqx7"
Sep 17 20:50:25.308157 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:50:25.308122    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-metrics-tls\") pod \"dns-default-78ncn\" (UID: \"ae0c2874-4c74-491b-8fa4-4ff0fcc84c00\") " pod="openshift-dns/dns-default-78ncn"
Sep 17 20:50:25.308614 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:50:25.308271    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:50:25.308614 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:50:25.308331    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-metrics-tls podName:ae0c2874-4c74-491b-8fa4-4ff0fcc84c00 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:57.308312675 +0000 UTC m=+98.919045903 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-metrics-tls") pod "dns-default-78ncn" (UID: "ae0c2874-4c74-491b-8fa4-4ff0fcc84c00") : secret "dns-default-metrics-tls" not found
Sep 17 20:50:26.113049 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:50:26.113005    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-metrics-certs\") pod \"network-metrics-daemon-sknnf\" (UID: \"8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7\") " pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:50:26.113240 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:50:26.113150    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 20:50:26.113240 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:50:26.113227    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-metrics-certs podName:8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:30.113195129 +0000 UTC m=+131.723928340 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-metrics-certs") pod "network-metrics-daemon-sknnf" (UID: "8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7") : secret "metrics-daemon-secret" not found
Sep 17 20:50:32.659378 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:50:32.659349    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9tkmw"
Sep 17 20:50:57.325385 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:50:57.325357    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-metrics-tls\") pod \"dns-default-78ncn\" (UID: \"ae0c2874-4c74-491b-8fa4-4ff0fcc84c00\") " pod="openshift-dns/dns-default-78ncn"
Sep 17 20:50:57.325633 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:50:57.325454    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:50:57.325633 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:50:57.325511    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-metrics-tls podName:ae0c2874-4c74-491b-8fa4-4ff0fcc84c00 nodeName:}" failed. No retries permitted until 2026-09-17 20:52:01.325498321 +0000 UTC m=+162.936231529 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-metrics-tls") pod "dns-default-78ncn" (UID: "ae0c2874-4c74-491b-8fa4-4ff0fcc84c00") : secret "dns-default-metrics-tls" not found
Sep 17 20:51:15.713714 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.713682    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lk69f"]
Sep 17 20:51:15.716404 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.716389    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lk69f"
Sep 17 20:51:15.722777 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.722761    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:51:15.722888 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.722770    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 17 20:51:15.723892 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.723879    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-ztlq5\""
Sep 17 20:51:15.732187 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.732169    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lk69f"]
Sep 17 20:51:15.803406 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.803382    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-t2rlv"]
Sep 17 20:51:15.806079 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.806064    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-t2rlv"
Sep 17 20:51:15.809156 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.809133    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 20:51:15.810117 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.810096    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-9jjhn\""
Sep 17 20:51:15.810300 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.810280    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:51:15.810826 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.810797    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 20:51:15.811058 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.810985    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 20:51:15.811589 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.811573    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-8dg4p"]
Sep 17 20:51:15.814199 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.814180    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d97bd56b7-ghblg"]
Sep 17 20:51:15.814394 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.814290    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8dg4p"
Sep 17 20:51:15.816398 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.816378    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 20:51:15.816582 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.816568    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 20:51:15.816724 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.816691    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-6kn6h\""
Sep 17 20:51:15.816823 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.816805    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 20:51:15.816876 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.816861    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:15.820079 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.820062    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 20:51:15.820721 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.820706    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 20:51:15.820804 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.820750    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9cn4m\""
Sep 17 20:51:15.821288 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.821260    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 20:51:15.821554 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.821536    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 20:51:15.821919 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.821810    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:51:15.825125 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.825106    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-t2rlv"]
Sep 17 20:51:15.827989 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.827959    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 20:51:15.831072 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.831053    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-8dg4p"]
Sep 17 20:51:15.836076 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.836057    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d97bd56b7-ghblg"]
Sep 17 20:51:15.837512 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.837492    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4684t\" (UniqueName: \"kubernetes.io/projected/d19d48a3-03c2-4fe4-a052-8b7e87b0055a-kube-api-access-4684t\") pod \"volume-data-source-validator-5c4db8bdf9-lk69f\" (UID: \"d19d48a3-03c2-4fe4-a052-8b7e87b0055a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lk69f"
Sep 17 20:51:15.938572 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.938544    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4684t\" (UniqueName: \"kubernetes.io/projected/d19d48a3-03c2-4fe4-a052-8b7e87b0055a-kube-api-access-4684t\") pod \"volume-data-source-validator-5c4db8bdf9-lk69f\" (UID: \"d19d48a3-03c2-4fe4-a052-8b7e87b0055a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lk69f"
Sep 17 20:51:15.938727 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.938590    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1552a7fa-597f-406a-8c77-cca66297356d-installation-pull-secrets\") pod \"image-registry-6d97bd56b7-ghblg\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") " pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:15.938727 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.938693    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-bound-sa-token\") pod \"image-registry-6d97bd56b7-ghblg\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") " pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:15.938804 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.938731    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sksfj\" (UniqueName: \"kubernetes.io/projected/aa1dd337-94f0-4707-a264-577fa255c317-kube-api-access-sksfj\") pod \"service-ca-operator-57cdc4b768-8dg4p\" (UID: \"aa1dd337-94f0-4707-a264-577fa255c317\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8dg4p"
Sep 17 20:51:15.938804 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.938752    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gckm2\" (UniqueName: \"kubernetes.io/projected/ed6472a6-7979-43b0-a549-7ca5c5d8fb0f-kube-api-access-gckm2\") pod \"console-operator-fd4bb46c-t2rlv\" (UID: \"ed6472a6-7979-43b0-a549-7ca5c5d8fb0f\") " pod="openshift-console-operator/console-operator-fd4bb46c-t2rlv"
Sep 17 20:51:15.938874 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.938824    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1552a7fa-597f-406a-8c77-cca66297356d-registry-certificates\") pod \"image-registry-6d97bd56b7-ghblg\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") " pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:15.938874 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.938855    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed6472a6-7979-43b0-a549-7ca5c5d8fb0f-trusted-ca\") pod \"console-operator-fd4bb46c-t2rlv\" (UID: \"ed6472a6-7979-43b0-a549-7ca5c5d8fb0f\") " pod="openshift-console-operator/console-operator-fd4bb46c-t2rlv"
Sep 17 20:51:15.938874 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.938871    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa1dd337-94f0-4707-a264-577fa255c317-config\") pod \"service-ca-operator-57cdc4b768-8dg4p\" (UID: \"aa1dd337-94f0-4707-a264-577fa255c317\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8dg4p"
Sep 17 20:51:15.938974 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.938914    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ed6472a6-7979-43b0-a549-7ca5c5d8fb0f-serving-cert\") pod \"console-operator-fd4bb46c-t2rlv\" (UID: \"ed6472a6-7979-43b0-a549-7ca5c5d8fb0f\") " pod="openshift-console-operator/console-operator-fd4bb46c-t2rlv"
Sep 17 20:51:15.938974 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.938949    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1552a7fa-597f-406a-8c77-cca66297356d-ca-trust-extracted\") pod \"image-registry-6d97bd56b7-ghblg\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") " pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:15.939043 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.938978    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1552a7fa-597f-406a-8c77-cca66297356d-trusted-ca\") pod \"image-registry-6d97bd56b7-ghblg\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") " pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:15.939043 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.939009    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa1dd337-94f0-4707-a264-577fa255c317-serving-cert\") pod \"service-ca-operator-57cdc4b768-8dg4p\" (UID: \"aa1dd337-94f0-4707-a264-577fa255c317\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8dg4p"
Sep 17 20:51:15.939043 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.939027    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1552a7fa-597f-406a-8c77-cca66297356d-image-registry-private-configuration\") pod \"image-registry-6d97bd56b7-ghblg\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") " pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:15.939139 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.939046    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkkcr\" (UniqueName: \"kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-kube-api-access-nkkcr\") pod \"image-registry-6d97bd56b7-ghblg\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") " pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:15.939139 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.939073    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-registry-tls\") pod \"image-registry-6d97bd56b7-ghblg\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") " pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:15.939139 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.939110    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ed6472a6-7979-43b0-a549-7ca5c5d8fb0f-config\") pod \"console-operator-fd4bb46c-t2rlv\" (UID: \"ed6472a6-7979-43b0-a549-7ca5c5d8fb0f\") " pod="openshift-console-operator/console-operator-fd4bb46c-t2rlv"
Sep 17 20:51:15.949878 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:15.949853    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4684t\" (UniqueName: \"kubernetes.io/projected/d19d48a3-03c2-4fe4-a052-8b7e87b0055a-kube-api-access-4684t\") pod \"volume-data-source-validator-5c4db8bdf9-lk69f\" (UID: \"d19d48a3-03c2-4fe4-a052-8b7e87b0055a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lk69f"
Sep 17 20:51:16.024976 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.024900    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lk69f"
Sep 17 20:51:16.040001 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.039981    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1552a7fa-597f-406a-8c77-cca66297356d-installation-pull-secrets\") pod \"image-registry-6d97bd56b7-ghblg\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") " pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:16.040092 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.040008    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-bound-sa-token\") pod \"image-registry-6d97bd56b7-ghblg\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") " pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:16.040092 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.040028    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sksfj\" (UniqueName: \"kubernetes.io/projected/aa1dd337-94f0-4707-a264-577fa255c317-kube-api-access-sksfj\") pod \"service-ca-operator-57cdc4b768-8dg4p\" (UID: \"aa1dd337-94f0-4707-a264-577fa255c317\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8dg4p"
Sep 17 20:51:16.040092 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.040046    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gckm2\" (UniqueName: \"kubernetes.io/projected/ed6472a6-7979-43b0-a549-7ca5c5d8fb0f-kube-api-access-gckm2\") pod \"console-operator-fd4bb46c-t2rlv\" (UID: \"ed6472a6-7979-43b0-a549-7ca5c5d8fb0f\") " pod="openshift-console-operator/console-operator-fd4bb46c-t2rlv"
Sep 17 20:51:16.040092 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.040061    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1552a7fa-597f-406a-8c77-cca66297356d-registry-certificates\") pod \"image-registry-6d97bd56b7-ghblg\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") " pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:16.040322 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.040273    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed6472a6-7979-43b0-a549-7ca5c5d8fb0f-trusted-ca\") pod \"console-operator-fd4bb46c-t2rlv\" (UID: \"ed6472a6-7979-43b0-a549-7ca5c5d8fb0f\") " pod="openshift-console-operator/console-operator-fd4bb46c-t2rlv"
Sep 17 20:51:16.040322 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.040312    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa1dd337-94f0-4707-a264-577fa255c317-config\") pod \"service-ca-operator-57cdc4b768-8dg4p\" (UID: \"aa1dd337-94f0-4707-a264-577fa255c317\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8dg4p"
Sep 17 20:51:16.040420 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.040354    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ed6472a6-7979-43b0-a549-7ca5c5d8fb0f-serving-cert\") pod \"console-operator-fd4bb46c-t2rlv\" (UID: \"ed6472a6-7979-43b0-a549-7ca5c5d8fb0f\") " pod="openshift-console-operator/console-operator-fd4bb46c-t2rlv"
Sep 17 20:51:16.040420 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.040379    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1552a7fa-597f-406a-8c77-cca66297356d-ca-trust-extracted\") pod \"image-registry-6d97bd56b7-ghblg\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") " pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:16.040420 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.040405    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1552a7fa-597f-406a-8c77-cca66297356d-trusted-ca\") pod \"image-registry-6d97bd56b7-ghblg\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") " pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:16.040558 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.040432    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa1dd337-94f0-4707-a264-577fa255c317-serving-cert\") pod \"service-ca-operator-57cdc4b768-8dg4p\" (UID: \"aa1dd337-94f0-4707-a264-577fa255c317\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8dg4p"
Sep 17 20:51:16.040558 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.040458    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1552a7fa-597f-406a-8c77-cca66297356d-image-registry-private-configuration\") pod \"image-registry-6d97bd56b7-ghblg\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") " pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:16.040558 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.040483    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkkcr\" (UniqueName: \"kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-kube-api-access-nkkcr\") pod \"image-registry-6d97bd56b7-ghblg\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") " pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:16.040558 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.040515    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-registry-tls\") pod \"image-registry-6d97bd56b7-ghblg\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") " pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:16.040558 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.040542    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1552a7fa-597f-406a-8c77-cca66297356d-registry-certificates\") pod \"image-registry-6d97bd56b7-ghblg\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") " pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:16.040558 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.040554    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ed6472a6-7979-43b0-a549-7ca5c5d8fb0f-config\") pod \"console-operator-fd4bb46c-t2rlv\" (UID: \"ed6472a6-7979-43b0-a549-7ca5c5d8fb0f\") " pod="openshift-console-operator/console-operator-fd4bb46c-t2rlv"
Sep 17 20:51:16.041402 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.041033    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa1dd337-94f0-4707-a264-577fa255c317-config\") pod \"service-ca-operator-57cdc4b768-8dg4p\" (UID: \"aa1dd337-94f0-4707-a264-577fa255c317\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8dg4p"
Sep 17 20:51:16.041402 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:16.041075    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:51:16.041402 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:16.041093    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d97bd56b7-ghblg: secret "image-registry-tls" not found
Sep 17 20:51:16.041402 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.041131    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1552a7fa-597f-406a-8c77-cca66297356d-ca-trust-extracted\") pod \"image-registry-6d97bd56b7-ghblg\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") " pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:16.041402 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:16.041148    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-registry-tls podName:1552a7fa-597f-406a-8c77-cca66297356d nodeName:}" failed. No retries permitted until 2026-09-17 20:51:16.541129948 +0000 UTC m=+118.151863162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-registry-tls") pod "image-registry-6d97bd56b7-ghblg" (UID: "1552a7fa-597f-406a-8c77-cca66297356d") : secret "image-registry-tls" not found
Sep 17 20:51:16.041402 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.041246    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ed6472a6-7979-43b0-a549-7ca5c5d8fb0f-config\") pod \"console-operator-fd4bb46c-t2rlv\" (UID: \"ed6472a6-7979-43b0-a549-7ca5c5d8fb0f\") " pod="openshift-console-operator/console-operator-fd4bb46c-t2rlv"
Sep 17 20:51:16.041402 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.041349    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed6472a6-7979-43b0-a549-7ca5c5d8fb0f-trusted-ca\") pod \"console-operator-fd4bb46c-t2rlv\" (UID: \"ed6472a6-7979-43b0-a549-7ca5c5d8fb0f\") " pod="openshift-console-operator/console-operator-fd4bb46c-t2rlv"
Sep 17 20:51:16.041870 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.041356    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1552a7fa-597f-406a-8c77-cca66297356d-trusted-ca\") pod \"image-registry-6d97bd56b7-ghblg\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") " pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:16.043045 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.043024    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1552a7fa-597f-406a-8c77-cca66297356d-image-registry-private-configuration\") pod \"image-registry-6d97bd56b7-ghblg\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") " pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:16.043268 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.043251    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa1dd337-94f0-4707-a264-577fa255c317-serving-cert\") pod \"service-ca-operator-57cdc4b768-8dg4p\" (UID: \"aa1dd337-94f0-4707-a264-577fa255c317\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8dg4p"
Sep 17 20:51:16.043337 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.043280    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1552a7fa-597f-406a-8c77-cca66297356d-installation-pull-secrets\") pod \"image-registry-6d97bd56b7-ghblg\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") " pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:16.043665 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.043651    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ed6472a6-7979-43b0-a549-7ca5c5d8fb0f-serving-cert\") pod \"console-operator-fd4bb46c-t2rlv\" (UID: \"ed6472a6-7979-43b0-a549-7ca5c5d8fb0f\") " pod="openshift-console-operator/console-operator-fd4bb46c-t2rlv"
Sep 17 20:51:16.049581 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.049560    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-bound-sa-token\") pod \"image-registry-6d97bd56b7-ghblg\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") " pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:16.049823 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.049805    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sksfj\" (UniqueName: \"kubernetes.io/projected/aa1dd337-94f0-4707-a264-577fa255c317-kube-api-access-sksfj\") pod \"service-ca-operator-57cdc4b768-8dg4p\" (UID: \"aa1dd337-94f0-4707-a264-577fa255c317\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8dg4p"
Sep 17 20:51:16.049894 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.049881    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkkcr\" (UniqueName: \"kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-kube-api-access-nkkcr\") pod \"image-registry-6d97bd56b7-ghblg\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") " pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:16.049937 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.049886    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gckm2\" (UniqueName: \"kubernetes.io/projected/ed6472a6-7979-43b0-a549-7ca5c5d8fb0f-kube-api-access-gckm2\") pod \"console-operator-fd4bb46c-t2rlv\" (UID: \"ed6472a6-7979-43b0-a549-7ca5c5d8fb0f\") " pod="openshift-console-operator/console-operator-fd4bb46c-t2rlv"
Sep 17 20:51:16.117686 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.117653    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-t2rlv"
Sep 17 20:51:16.125733 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.125711    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8dg4p"
Sep 17 20:51:16.291582 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.291504    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lk69f"]
Sep 17 20:51:16.294374 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:51:16.294343    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd19d48a3_03c2_4fe4_a052_8b7e87b0055a.slice/crio-b2bd2d815b893284bb8fed58ea799b4f154c2726effe0b7841e584890a308560 WatchSource:0}: Error finding container b2bd2d815b893284bb8fed58ea799b4f154c2726effe0b7841e584890a308560: Status 404 returned error can't find the container with id b2bd2d815b893284bb8fed58ea799b4f154c2726effe0b7841e584890a308560
Sep 17 20:51:16.392518 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.392488    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-t2rlv"]
Sep 17 20:51:16.396241 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:51:16.396186    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded6472a6_7979_43b0_a549_7ca5c5d8fb0f.slice/crio-bf8a20dff3153138e6e1eca824dec3385fbbfb1c613737e03f8548967e689f17 WatchSource:0}: Error finding container bf8a20dff3153138e6e1eca824dec3385fbbfb1c613737e03f8548967e689f17: Status 404 returned error can't find the container with id bf8a20dff3153138e6e1eca824dec3385fbbfb1c613737e03f8548967e689f17
Sep 17 20:51:16.408323 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.408301    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-8dg4p"]
Sep 17 20:51:16.411302 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:51:16.411270    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa1dd337_94f0_4707_a264_577fa255c317.slice/crio-408710f80950856bf31151bedbeb08b4648f018e4e3a45db5412aff603609e83 WatchSource:0}: Error finding container 408710f80950856bf31151bedbeb08b4648f018e4e3a45db5412aff603609e83: Status 404 returned error can't find the container with id 408710f80950856bf31151bedbeb08b4648f018e4e3a45db5412aff603609e83
Sep 17 20:51:16.547642 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.547546    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-registry-tls\") pod \"image-registry-6d97bd56b7-ghblg\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") " pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:16.547772 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:16.547702    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:51:16.547772 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:16.547721    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d97bd56b7-ghblg: secret "image-registry-tls" not found
Sep 17 20:51:16.547843 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:16.547809    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-registry-tls podName:1552a7fa-597f-406a-8c77-cca66297356d nodeName:}" failed. No retries permitted until 2026-09-17 20:51:17.547791769 +0000 UTC m=+119.158524976 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-registry-tls") pod "image-registry-6d97bd56b7-ghblg" (UID: "1552a7fa-597f-406a-8c77-cca66297356d") : secret "image-registry-tls" not found
Sep 17 20:51:16.834190 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.834104    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-t2rlv" event={"ID":"ed6472a6-7979-43b0-a549-7ca5c5d8fb0f","Type":"ContainerStarted","Data":"bf8a20dff3153138e6e1eca824dec3385fbbfb1c613737e03f8548967e689f17"}
Sep 17 20:51:16.835128 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.835109    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lk69f" event={"ID":"d19d48a3-03c2-4fe4-a052-8b7e87b0055a","Type":"ContainerStarted","Data":"b2bd2d815b893284bb8fed58ea799b4f154c2726effe0b7841e584890a308560"}
Sep 17 20:51:16.836066 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:16.836046    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8dg4p" event={"ID":"aa1dd337-94f0-4707-a264-577fa255c317","Type":"ContainerStarted","Data":"408710f80950856bf31151bedbeb08b4648f018e4e3a45db5412aff603609e83"}
Sep 17 20:51:17.554643 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:17.554604    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-registry-tls\") pod \"image-registry-6d97bd56b7-ghblg\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") " pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:17.554833 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:17.554788    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:51:17.554833 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:17.554815    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d97bd56b7-ghblg: secret "image-registry-tls" not found
Sep 17 20:51:17.554952 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:17.554901    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-registry-tls podName:1552a7fa-597f-406a-8c77-cca66297356d nodeName:}" failed. No retries permitted until 2026-09-17 20:51:19.554876611 +0000 UTC m=+121.165609843 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-registry-tls") pod "image-registry-6d97bd56b7-ghblg" (UID: "1552a7fa-597f-406a-8c77-cca66297356d") : secret "image-registry-tls" not found
Sep 17 20:51:17.841160 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:17.841074    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lk69f" event={"ID":"d19d48a3-03c2-4fe4-a052-8b7e87b0055a","Type":"ContainerStarted","Data":"31ff27e3efbcd2f271980a2ee58c8d9707d5a765b963bd70b782ac91c73e207f"}
Sep 17 20:51:17.858269 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:17.858196    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lk69f" podStartSLOduration=1.5338762350000001 podStartE2EDuration="2.858181606s" totalImagesPullingTime="1.324305371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:17.856770163 +0000 UTC m=+119.467503392" watchObservedRunningTime="2026-09-17 20:51:17.858181606 +0000 UTC m=+119.468914836"
Sep 17 20:51:18.846824 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:18.846786    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8dg4p" event={"ID":"aa1dd337-94f0-4707-a264-577fa255c317","Type":"ContainerStarted","Data":"96dcc6984eb8ab9294f22fc96dcfbd72e4ee09c513ab5b5ca8a4159b0ebc10d8"}
Sep 17 20:51:18.849032 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:18.848998    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-t2rlv" event={"ID":"ed6472a6-7979-43b0-a549-7ca5c5d8fb0f","Type":"ContainerStarted","Data":"84d14e90a16f3c766e3af3d74434ad1e0e2acebe51751da8086e5944fee0034d"}
Sep 17 20:51:18.849237 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:18.849182    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-t2rlv"
Sep 17 20:51:18.850797 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:18.850765    2566 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-t2rlv container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.133.0.9:8443/readyz\": dial tcp 10.133.0.9:8443: connect: connection refused" start-of-body=
Sep 17 20:51:18.850886 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:18.850824    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-t2rlv" podUID="ed6472a6-7979-43b0-a549-7ca5c5d8fb0f" containerName="console-operator" probeResult="failure" output="Get \"https://10.133.0.9:8443/readyz\": dial tcp 10.133.0.9:8443: connect: connection refused"
Sep 17 20:51:18.871235 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:18.871161    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8dg4p" podStartSLOduration=1.537098835 podStartE2EDuration="3.871143838s" totalImagesPullingTime="2.334045003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:18.869883547 +0000 UTC m=+120.480616776" watchObservedRunningTime="2026-09-17 20:51:18.871143838 +0000 UTC m=+120.481877155"
Sep 17 20:51:18.891518 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:18.891471    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-t2rlv" podStartSLOduration=1.5459738669999998 podStartE2EDuration="3.891454136s" totalImagesPullingTime="2.345480269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:18.890450916 +0000 UTC m=+120.501184146" watchObservedRunningTime="2026-09-17 20:51:18.891454136 +0000 UTC m=+120.502187365"
Sep 17 20:51:19.482328 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:19.482300    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-7mfg4"]
Sep 17 20:51:19.485322 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:19.485294    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7mfg4"
Sep 17 20:51:19.491400 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:19.491376    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-j2fh9\""
Sep 17 20:51:19.495573 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:19.495555    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 20:51:19.498423 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:19.498389    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 20:51:19.514402 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:19.514375    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-7mfg4"]
Sep 17 20:51:19.572802 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:19.572770    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmz8l\" (UniqueName: \"kubernetes.io/projected/b92f4da1-f4c2-428b-ad13-8892f296be13-kube-api-access-cmz8l\") pod \"migrator-76794cbd6c-7mfg4\" (UID: \"b92f4da1-f4c2-428b-ad13-8892f296be13\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7mfg4"
Sep 17 20:51:19.572941 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:19.572830    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-registry-tls\") pod \"image-registry-6d97bd56b7-ghblg\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") " pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:19.572941 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:19.572918    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:51:19.572941 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:19.572928    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d97bd56b7-ghblg: secret "image-registry-tls" not found
Sep 17 20:51:19.573032 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:19.572979    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-registry-tls podName:1552a7fa-597f-406a-8c77-cca66297356d nodeName:}" failed. No retries permitted until 2026-09-17 20:51:23.572966935 +0000 UTC m=+125.183700143 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-registry-tls") pod "image-registry-6d97bd56b7-ghblg" (UID: "1552a7fa-597f-406a-8c77-cca66297356d") : secret "image-registry-tls" not found
Sep 17 20:51:19.673612 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:19.673581    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmz8l\" (UniqueName: \"kubernetes.io/projected/b92f4da1-f4c2-428b-ad13-8892f296be13-kube-api-access-cmz8l\") pod \"migrator-76794cbd6c-7mfg4\" (UID: \"b92f4da1-f4c2-428b-ad13-8892f296be13\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7mfg4"
Sep 17 20:51:19.683778 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:19.683752    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmz8l\" (UniqueName: \"kubernetes.io/projected/b92f4da1-f4c2-428b-ad13-8892f296be13-kube-api-access-cmz8l\") pod \"migrator-76794cbd6c-7mfg4\" (UID: \"b92f4da1-f4c2-428b-ad13-8892f296be13\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7mfg4"
Sep 17 20:51:19.793594 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:19.793529    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7mfg4"
Sep 17 20:51:19.855014 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:19.854981    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t2rlv_ed6472a6-7979-43b0-a549-7ca5c5d8fb0f/console-operator/0.log"
Sep 17 20:51:19.855414 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:19.855031    2566 generic.go:356] "Generic (PLEG): container finished" podID="ed6472a6-7979-43b0-a549-7ca5c5d8fb0f" containerID="84d14e90a16f3c766e3af3d74434ad1e0e2acebe51751da8086e5944fee0034d" exitCode=255
Sep 17 20:51:19.855414 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:19.855228    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-t2rlv" event={"ID":"ed6472a6-7979-43b0-a549-7ca5c5d8fb0f","Type":"ContainerDied","Data":"84d14e90a16f3c766e3af3d74434ad1e0e2acebe51751da8086e5944fee0034d"}
Sep 17 20:51:19.855719 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:19.855695    2566 scope.go:117] "RemoveContainer" containerID="84d14e90a16f3c766e3af3d74434ad1e0e2acebe51751da8086e5944fee0034d"
Sep 17 20:51:20.063645 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.063571    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-7mfg4"]
Sep 17 20:51:20.066318 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:51:20.066295    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb92f4da1_f4c2_428b_ad13_8892f296be13.slice/crio-37e9cb4e4b04dc71e8645705641f9c850c934e4c4088e7fdee9ad389c7eef468 WatchSource:0}: Error finding container 37e9cb4e4b04dc71e8645705641f9c850c934e4c4088e7fdee9ad389c7eef468: Status 404 returned error can't find the container with id 37e9cb4e4b04dc71e8645705641f9c850c934e4c4088e7fdee9ad389c7eef468
Sep 17 20:51:20.213786 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.213759    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7m89j"]
Sep 17 20:51:20.216975 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.216958    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7m89j"
Sep 17 20:51:20.219692 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.219672    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 20:51:20.219844 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.219826    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5hgxg\""
Sep 17 20:51:20.219991 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.219971    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 20:51:20.220054 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.220028    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 20:51:20.220291 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.220276    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 20:51:20.230859 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.230838    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7m89j"]
Sep 17 20:51:20.277318 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.277296    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/102ec935-a1f0-4e2b-84ae-8832e11a1033-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7m89j\" (UID: \"102ec935-a1f0-4e2b-84ae-8832e11a1033\") " pod="openshift-insights/insights-runtime-extractor-7m89j"
Sep 17 20:51:20.277414 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.277338    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/102ec935-a1f0-4e2b-84ae-8832e11a1033-data-volume\") pod \"insights-runtime-extractor-7m89j\" (UID: \"102ec935-a1f0-4e2b-84ae-8832e11a1033\") " pod="openshift-insights/insights-runtime-extractor-7m89j"
Sep 17 20:51:20.277414 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.277373    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/102ec935-a1f0-4e2b-84ae-8832e11a1033-crio-socket\") pod \"insights-runtime-extractor-7m89j\" (UID: \"102ec935-a1f0-4e2b-84ae-8832e11a1033\") " pod="openshift-insights/insights-runtime-extractor-7m89j"
Sep 17 20:51:20.277414 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.277395    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw8vj\" (UniqueName: \"kubernetes.io/projected/102ec935-a1f0-4e2b-84ae-8832e11a1033-kube-api-access-jw8vj\") pod \"insights-runtime-extractor-7m89j\" (UID: \"102ec935-a1f0-4e2b-84ae-8832e11a1033\") " pod="openshift-insights/insights-runtime-extractor-7m89j"
Sep 17 20:51:20.277414 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.277413    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/102ec935-a1f0-4e2b-84ae-8832e11a1033-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7m89j\" (UID: \"102ec935-a1f0-4e2b-84ae-8832e11a1033\") " pod="openshift-insights/insights-runtime-extractor-7m89j"
Sep 17 20:51:20.377653 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.377599    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/102ec935-a1f0-4e2b-84ae-8832e11a1033-data-volume\") pod \"insights-runtime-extractor-7m89j\" (UID: \"102ec935-a1f0-4e2b-84ae-8832e11a1033\") " pod="openshift-insights/insights-runtime-extractor-7m89j"
Sep 17 20:51:20.377653 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.377632    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/102ec935-a1f0-4e2b-84ae-8832e11a1033-crio-socket\") pod \"insights-runtime-extractor-7m89j\" (UID: \"102ec935-a1f0-4e2b-84ae-8832e11a1033\") " pod="openshift-insights/insights-runtime-extractor-7m89j"
Sep 17 20:51:20.377653 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.377649    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw8vj\" (UniqueName: \"kubernetes.io/projected/102ec935-a1f0-4e2b-84ae-8832e11a1033-kube-api-access-jw8vj\") pod \"insights-runtime-extractor-7m89j\" (UID: \"102ec935-a1f0-4e2b-84ae-8832e11a1033\") " pod="openshift-insights/insights-runtime-extractor-7m89j"
Sep 17 20:51:20.377864 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.377667    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/102ec935-a1f0-4e2b-84ae-8832e11a1033-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7m89j\" (UID: \"102ec935-a1f0-4e2b-84ae-8832e11a1033\") " pod="openshift-insights/insights-runtime-extractor-7m89j"
Sep 17 20:51:20.377864 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.377725    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/102ec935-a1f0-4e2b-84ae-8832e11a1033-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7m89j\" (UID: \"102ec935-a1f0-4e2b-84ae-8832e11a1033\") " pod="openshift-insights/insights-runtime-extractor-7m89j"
Sep 17 20:51:20.377864 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.377733    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/102ec935-a1f0-4e2b-84ae-8832e11a1033-crio-socket\") pod \"insights-runtime-extractor-7m89j\" (UID: \"102ec935-a1f0-4e2b-84ae-8832e11a1033\") " pod="openshift-insights/insights-runtime-extractor-7m89j"
Sep 17 20:51:20.378014 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.377945    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/102ec935-a1f0-4e2b-84ae-8832e11a1033-data-volume\") pod \"insights-runtime-extractor-7m89j\" (UID: \"102ec935-a1f0-4e2b-84ae-8832e11a1033\") " pod="openshift-insights/insights-runtime-extractor-7m89j"
Sep 17 20:51:20.380244 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.380225    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 20:51:20.380351 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.380260    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 20:51:20.387947 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:20.387915    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:51:20.388016 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:20.387980    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/102ec935-a1f0-4e2b-84ae-8832e11a1033-kube-rbac-proxy-cm podName:102ec935-a1f0-4e2b-84ae-8832e11a1033 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:20.887961411 +0000 UTC m=+122.498694619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/102ec935-a1f0-4e2b-84ae-8832e11a1033-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7m89j" (UID: "102ec935-a1f0-4e2b-84ae-8832e11a1033") : configmap "kube-rbac-proxy" not found
Sep 17 20:51:20.388016 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:20.387990    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:20.388113 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:20.388038    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/102ec935-a1f0-4e2b-84ae-8832e11a1033-insights-runtime-extractor-tls podName:102ec935-a1f0-4e2b-84ae-8832e11a1033 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:20.888026041 +0000 UTC m=+122.498759253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/102ec935-a1f0-4e2b-84ae-8832e11a1033-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7m89j" (UID: "102ec935-a1f0-4e2b-84ae-8832e11a1033") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:20.388113 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.388062    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 20:51:20.398046 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.398028    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 20:51:20.408105 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.408081    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw8vj\" (UniqueName: \"kubernetes.io/projected/102ec935-a1f0-4e2b-84ae-8832e11a1033-kube-api-access-jw8vj\") pod \"insights-runtime-extractor-7m89j\" (UID: \"102ec935-a1f0-4e2b-84ae-8832e11a1033\") " pod="openshift-insights/insights-runtime-extractor-7m89j"
Sep 17 20:51:20.859912 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.859885    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t2rlv_ed6472a6-7979-43b0-a549-7ca5c5d8fb0f/console-operator/1.log"
Sep 17 20:51:20.860395 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.860274    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t2rlv_ed6472a6-7979-43b0-a549-7ca5c5d8fb0f/console-operator/0.log"
Sep 17 20:51:20.860395 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.860306    2566 generic.go:356] "Generic (PLEG): container finished" podID="ed6472a6-7979-43b0-a549-7ca5c5d8fb0f" containerID="5686d22825767e32e9113041d4700873cb9e80dd8acb1dbf59c9aa0bee3e9f4f" exitCode=255
Sep 17 20:51:20.860487 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.860387    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-t2rlv" event={"ID":"ed6472a6-7979-43b0-a549-7ca5c5d8fb0f","Type":"ContainerDied","Data":"5686d22825767e32e9113041d4700873cb9e80dd8acb1dbf59c9aa0bee3e9f4f"}
Sep 17 20:51:20.860487 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.860482    2566 scope.go:117] "RemoveContainer" containerID="84d14e90a16f3c766e3af3d74434ad1e0e2acebe51751da8086e5944fee0034d"
Sep 17 20:51:20.860727 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.860709    2566 scope.go:117] "RemoveContainer" containerID="5686d22825767e32e9113041d4700873cb9e80dd8acb1dbf59c9aa0bee3e9f4f"
Sep 17 20:51:20.860912 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:20.860891    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-t2rlv_openshift-console-operator(ed6472a6-7979-43b0-a549-7ca5c5d8fb0f)\"" pod="openshift-console-operator/console-operator-fd4bb46c-t2rlv" podUID="ed6472a6-7979-43b0-a549-7ca5c5d8fb0f"
Sep 17 20:51:20.861485 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.861462    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7mfg4" event={"ID":"b92f4da1-f4c2-428b-ad13-8892f296be13","Type":"ContainerStarted","Data":"37e9cb4e4b04dc71e8645705641f9c850c934e4c4088e7fdee9ad389c7eef468"}
Sep 17 20:51:20.982181 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.982139    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/102ec935-a1f0-4e2b-84ae-8832e11a1033-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7m89j\" (UID: \"102ec935-a1f0-4e2b-84ae-8832e11a1033\") " pod="openshift-insights/insights-runtime-extractor-7m89j"
Sep 17 20:51:20.982388 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:20.982257    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/102ec935-a1f0-4e2b-84ae-8832e11a1033-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7m89j\" (UID: \"102ec935-a1f0-4e2b-84ae-8832e11a1033\") " pod="openshift-insights/insights-runtime-extractor-7m89j"
Sep 17 20:51:20.982388 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:20.982275    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:51:20.982388 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:20.982362    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/102ec935-a1f0-4e2b-84ae-8832e11a1033-kube-rbac-proxy-cm podName:102ec935-a1f0-4e2b-84ae-8832e11a1033 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:21.982331855 +0000 UTC m=+123.593065086 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/102ec935-a1f0-4e2b-84ae-8832e11a1033-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7m89j" (UID: "102ec935-a1f0-4e2b-84ae-8832e11a1033") : configmap "kube-rbac-proxy" not found
Sep 17 20:51:20.982604 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:20.982420    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:20.982604 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:20.982499    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/102ec935-a1f0-4e2b-84ae-8832e11a1033-insights-runtime-extractor-tls podName:102ec935-a1f0-4e2b-84ae-8832e11a1033 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:21.982480068 +0000 UTC m=+123.593213300 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/102ec935-a1f0-4e2b-84ae-8832e11a1033-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7m89j" (UID: "102ec935-a1f0-4e2b-84ae-8832e11a1033") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:21.866025 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:21.865997    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t2rlv_ed6472a6-7979-43b0-a549-7ca5c5d8fb0f/console-operator/1.log"
Sep 17 20:51:21.866491 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:21.866378    2566 scope.go:117] "RemoveContainer" containerID="5686d22825767e32e9113041d4700873cb9e80dd8acb1dbf59c9aa0bee3e9f4f"
Sep 17 20:51:21.866583 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:21.866562    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-t2rlv_openshift-console-operator(ed6472a6-7979-43b0-a549-7ca5c5d8fb0f)\"" pod="openshift-console-operator/console-operator-fd4bb46c-t2rlv" podUID="ed6472a6-7979-43b0-a549-7ca5c5d8fb0f"
Sep 17 20:51:21.867934 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:21.867906    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7mfg4" event={"ID":"b92f4da1-f4c2-428b-ad13-8892f296be13","Type":"ContainerStarted","Data":"c0f83b2519b7ff70cff488ffd187703b84c5551ea77223d1ae80e02059bc8277"}
Sep 17 20:51:21.868011 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:21.867950    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7mfg4" event={"ID":"b92f4da1-f4c2-428b-ad13-8892f296be13","Type":"ContainerStarted","Data":"47388dcf793dcb05b2698a9481d734aa40d75ebb945041419cd2ce9c61574ead"}
Sep 17 20:51:21.907084 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:21.907043    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7mfg4" podStartSLOduration=1.806055926 podStartE2EDuration="2.907029568s" totalImagesPullingTime="1.100973642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:21.906320858 +0000 UTC m=+123.517054086" watchObservedRunningTime="2026-09-17 20:51:21.907029568 +0000 UTC m=+123.517762838"
Sep 17 20:51:21.989757 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:21.989724    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/102ec935-a1f0-4e2b-84ae-8832e11a1033-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7m89j\" (UID: \"102ec935-a1f0-4e2b-84ae-8832e11a1033\") " pod="openshift-insights/insights-runtime-extractor-7m89j"
Sep 17 20:51:21.989899 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:21.989808    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/102ec935-a1f0-4e2b-84ae-8832e11a1033-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7m89j\" (UID: \"102ec935-a1f0-4e2b-84ae-8832e11a1033\") " pod="openshift-insights/insights-runtime-extractor-7m89j"
Sep 17 20:51:21.989899 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:21.989846    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:51:21.989979 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:21.989912    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/102ec935-a1f0-4e2b-84ae-8832e11a1033-kube-rbac-proxy-cm podName:102ec935-a1f0-4e2b-84ae-8832e11a1033 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:23.989892415 +0000 UTC m=+125.600625642 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/102ec935-a1f0-4e2b-84ae-8832e11a1033-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7m89j" (UID: "102ec935-a1f0-4e2b-84ae-8832e11a1033") : configmap "kube-rbac-proxy" not found
Sep 17 20:51:21.989979 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:21.989922    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:21.989979 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:21.989975    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/102ec935-a1f0-4e2b-84ae-8832e11a1033-insights-runtime-extractor-tls podName:102ec935-a1f0-4e2b-84ae-8832e11a1033 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:23.989959681 +0000 UTC m=+125.600692902 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/102ec935-a1f0-4e2b-84ae-8832e11a1033-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7m89j" (UID: "102ec935-a1f0-4e2b-84ae-8832e11a1033") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:22.773188 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:22.773160    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j7t6x_5f5d4daa-933c-47d6-b93a-6f83ed7ab23f/dns-node-resolver/0.log"
Sep 17 20:51:23.600367 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:23.600325    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-registry-tls\") pod \"image-registry-6d97bd56b7-ghblg\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") " pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:23.600769 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:23.600476    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:51:23.600769 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:23.600493    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d97bd56b7-ghblg: secret "image-registry-tls" not found
Sep 17 20:51:23.600769 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:23.600542    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-registry-tls podName:1552a7fa-597f-406a-8c77-cca66297356d nodeName:}" failed. No retries permitted until 2026-09-17 20:51:31.600528563 +0000 UTC m=+133.211261770 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-registry-tls") pod "image-registry-6d97bd56b7-ghblg" (UID: "1552a7fa-597f-406a-8c77-cca66297356d") : secret "image-registry-tls" not found
Sep 17 20:51:23.975633 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:23.975605    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j8n57_5e2bd987-6d8a-457b-ad67-f1e6597864e5/node-ca/0.log"
Sep 17 20:51:24.002875 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:24.002851    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/102ec935-a1f0-4e2b-84ae-8832e11a1033-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7m89j\" (UID: \"102ec935-a1f0-4e2b-84ae-8832e11a1033\") " pod="openshift-insights/insights-runtime-extractor-7m89j"
Sep 17 20:51:24.002942 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:24.002909    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/102ec935-a1f0-4e2b-84ae-8832e11a1033-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7m89j\" (UID: \"102ec935-a1f0-4e2b-84ae-8832e11a1033\") " pod="openshift-insights/insights-runtime-extractor-7m89j"
Sep 17 20:51:24.002983 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:24.002970    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:51:24.003025 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:24.002987    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:24.003063 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:24.003026    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/102ec935-a1f0-4e2b-84ae-8832e11a1033-kube-rbac-proxy-cm podName:102ec935-a1f0-4e2b-84ae-8832e11a1033 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:28.003012644 +0000 UTC m=+129.613745851 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/102ec935-a1f0-4e2b-84ae-8832e11a1033-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7m89j" (UID: "102ec935-a1f0-4e2b-84ae-8832e11a1033") : configmap "kube-rbac-proxy" not found
Sep 17 20:51:24.003063 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:24.003052    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/102ec935-a1f0-4e2b-84ae-8832e11a1033-insights-runtime-extractor-tls podName:102ec935-a1f0-4e2b-84ae-8832e11a1033 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:28.003038706 +0000 UTC m=+129.613771915 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/102ec935-a1f0-4e2b-84ae-8832e11a1033-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7m89j" (UID: "102ec935-a1f0-4e2b-84ae-8832e11a1033") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:24.974303 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:24.974276    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vnd5t_850f96d3-cdee-4f2e-8e36-f32466aa1bbb/serve-healthcheck-canary/0.log"
Sep 17 20:51:25.174443 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:25.174416    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-7mfg4_b92f4da1-f4c2-428b-ad13-8892f296be13/migrator/0.log"
Sep 17 20:51:25.374259 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:25.374189    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-7mfg4_b92f4da1-f4c2-428b-ad13-8892f296be13/graceful-termination/0.log"
Sep 17 20:51:26.117847 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:26.117813    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-t2rlv"
Sep 17 20:51:26.118228 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:26.118152    2566 scope.go:117] "RemoveContainer" containerID="5686d22825767e32e9113041d4700873cb9e80dd8acb1dbf59c9aa0bee3e9f4f"
Sep 17 20:51:26.118327 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:26.118310    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-t2rlv_openshift-console-operator(ed6472a6-7979-43b0-a549-7ca5c5d8fb0f)\"" pod="openshift-console-operator/console-operator-fd4bb46c-t2rlv" podUID="ed6472a6-7979-43b0-a549-7ca5c5d8fb0f"
Sep 17 20:51:28.028820 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:28.028783    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/102ec935-a1f0-4e2b-84ae-8832e11a1033-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7m89j\" (UID: \"102ec935-a1f0-4e2b-84ae-8832e11a1033\") " pod="openshift-insights/insights-runtime-extractor-7m89j"
Sep 17 20:51:28.029296 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:28.028853    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/102ec935-a1f0-4e2b-84ae-8832e11a1033-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7m89j\" (UID: \"102ec935-a1f0-4e2b-84ae-8832e11a1033\") " pod="openshift-insights/insights-runtime-extractor-7m89j"
Sep 17 20:51:28.029296 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:28.028886    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:51:28.029296 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:28.028953    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/102ec935-a1f0-4e2b-84ae-8832e11a1033-kube-rbac-proxy-cm podName:102ec935-a1f0-4e2b-84ae-8832e11a1033 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:36.028938734 +0000 UTC m=+137.639671941 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/102ec935-a1f0-4e2b-84ae-8832e11a1033-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7m89j" (UID: "102ec935-a1f0-4e2b-84ae-8832e11a1033") : configmap "kube-rbac-proxy" not found
Sep 17 20:51:28.029296 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:28.028995    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:28.029296 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:28.029057    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/102ec935-a1f0-4e2b-84ae-8832e11a1033-insights-runtime-extractor-tls podName:102ec935-a1f0-4e2b-84ae-8832e11a1033 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:36.029039645 +0000 UTC m=+137.639772854 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/102ec935-a1f0-4e2b-84ae-8832e11a1033-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7m89j" (UID: "102ec935-a1f0-4e2b-84ae-8832e11a1033") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:28.850119 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:28.850084    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-t2rlv"
Sep 17 20:51:28.850504 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:28.850487    2566 scope.go:117] "RemoveContainer" containerID="5686d22825767e32e9113041d4700873cb9e80dd8acb1dbf59c9aa0bee3e9f4f"
Sep 17 20:51:28.850689 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:28.850667    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-t2rlv_openshift-console-operator(ed6472a6-7979-43b0-a549-7ca5c5d8fb0f)\"" pod="openshift-console-operator/console-operator-fd4bb46c-t2rlv" podUID="ed6472a6-7979-43b0-a549-7ca5c5d8fb0f"
Sep 17 20:51:30.145128 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:30.145100    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-metrics-certs\") pod \"network-metrics-daemon-sknnf\" (UID: \"8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7\") " pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:51:30.145500 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:30.145246    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 20:51:30.145500 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:30.145301    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-metrics-certs podName:8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7 nodeName:}" failed. No retries permitted until 2026-09-17 20:53:32.145288654 +0000 UTC m=+253.756021861 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-metrics-certs") pod "network-metrics-daemon-sknnf" (UID: "8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7") : secret "metrics-daemon-secret" not found
Sep 17 20:51:31.655364 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:31.655330    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-registry-tls\") pod \"image-registry-6d97bd56b7-ghblg\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") " pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:31.657628 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:31.657608    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-registry-tls\") pod \"image-registry-6d97bd56b7-ghblg\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") " pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:31.733640 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:31.733595    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9cn4m\""
Sep 17 20:51:31.741832 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:31.741814    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:32.014520 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:32.014400    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d97bd56b7-ghblg"]
Sep 17 20:51:32.026854 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:51:32.026809    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1552a7fa_597f_406a_8c77_cca66297356d.slice/crio-9897121a05cbdc545367d7de09b8a582f7cb1829eb92bc69c6c86bf6283c0215 WatchSource:0}: Error finding container 9897121a05cbdc545367d7de09b8a582f7cb1829eb92bc69c6c86bf6283c0215: Status 404 returned error can't find the container with id 9897121a05cbdc545367d7de09b8a582f7cb1829eb92bc69c6c86bf6283c0215
Sep 17 20:51:32.907174 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:32.907137    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg" event={"ID":"1552a7fa-597f-406a-8c77-cca66297356d","Type":"ContainerStarted","Data":"ea8b815715f63c2be65ab3b71e0c10b3c840085e7d71e3f22e7959b3536be19c"}
Sep 17 20:51:32.907672 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:32.907194    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg" event={"ID":"1552a7fa-597f-406a-8c77-cca66297356d","Type":"ContainerStarted","Data":"9897121a05cbdc545367d7de09b8a582f7cb1829eb92bc69c6c86bf6283c0215"}
Sep 17 20:51:32.907672 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:32.907249    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:32.931002 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:32.930958    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg" podStartSLOduration=17.930943164 podStartE2EDuration="17.930943164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:32.929073855 +0000 UTC m=+134.539807084" watchObservedRunningTime="2026-09-17 20:51:32.930943164 +0000 UTC m=+134.541676393"
Sep 17 20:51:36.091058 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:36.091023    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/102ec935-a1f0-4e2b-84ae-8832e11a1033-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7m89j\" (UID: \"102ec935-a1f0-4e2b-84ae-8832e11a1033\") " pod="openshift-insights/insights-runtime-extractor-7m89j"
Sep 17 20:51:36.091510 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:36.091091    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/102ec935-a1f0-4e2b-84ae-8832e11a1033-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7m89j\" (UID: \"102ec935-a1f0-4e2b-84ae-8832e11a1033\") " pod="openshift-insights/insights-runtime-extractor-7m89j"
Sep 17 20:51:36.091510 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:36.091183    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:36.091510 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:36.091278    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/102ec935-a1f0-4e2b-84ae-8832e11a1033-insights-runtime-extractor-tls podName:102ec935-a1f0-4e2b-84ae-8832e11a1033 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:52.091260323 +0000 UTC m=+153.701993545 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/102ec935-a1f0-4e2b-84ae-8832e11a1033-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7m89j" (UID: "102ec935-a1f0-4e2b-84ae-8832e11a1033") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:36.091709 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:36.091692    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/102ec935-a1f0-4e2b-84ae-8832e11a1033-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7m89j\" (UID: \"102ec935-a1f0-4e2b-84ae-8832e11a1033\") " pod="openshift-insights/insights-runtime-extractor-7m89j"
Sep 17 20:51:43.454655 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:43.454623    2566 scope.go:117] "RemoveContainer" containerID="5686d22825767e32e9113041d4700873cb9e80dd8acb1dbf59c9aa0bee3e9f4f"
Sep 17 20:51:43.939796 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:43.939771    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t2rlv_ed6472a6-7979-43b0-a549-7ca5c5d8fb0f/console-operator/1.log"
Sep 17 20:51:43.939950 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:43.939843    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-t2rlv" event={"ID":"ed6472a6-7979-43b0-a549-7ca5c5d8fb0f","Type":"ContainerStarted","Data":"8cce842e80ce16811a0a8478beede3f88ec32377c288b59ca5d348e544745e65"}
Sep 17 20:51:43.940138 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:43.940114    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-t2rlv"
Sep 17 20:51:43.945361 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:43.945335    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-t2rlv"
Sep 17 20:51:46.120093 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:46.120065    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d97bd56b7-ghblg"]
Sep 17 20:51:52.108029 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:52.107989    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/102ec935-a1f0-4e2b-84ae-8832e11a1033-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7m89j\" (UID: \"102ec935-a1f0-4e2b-84ae-8832e11a1033\") " pod="openshift-insights/insights-runtime-extractor-7m89j"
Sep 17 20:51:52.110234 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:52.110202    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/102ec935-a1f0-4e2b-84ae-8832e11a1033-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7m89j\" (UID: \"102ec935-a1f0-4e2b-84ae-8832e11a1033\") " pod="openshift-insights/insights-runtime-extractor-7m89j"
Sep 17 20:51:52.328461 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:52.328429    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5hgxg\""
Sep 17 20:51:52.336553 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:52.336530    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7m89j"
Sep 17 20:51:52.611515 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:52.611480    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7m89j"]
Sep 17 20:51:52.615275 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:51:52.615240    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod102ec935_a1f0_4e2b_84ae_8832e11a1033.slice/crio-d212be5d0d358d99caac6ba7fd72a241086dd008ff4844255d873121f5367989 WatchSource:0}: Error finding container d212be5d0d358d99caac6ba7fd72a241086dd008ff4844255d873121f5367989: Status 404 returned error can't find the container with id d212be5d0d358d99caac6ba7fd72a241086dd008ff4844255d873121f5367989
Sep 17 20:51:52.969886 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:52.969854    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7m89j" event={"ID":"102ec935-a1f0-4e2b-84ae-8832e11a1033","Type":"ContainerStarted","Data":"f7b2223c260be9746aea72c55b67b4f237a47e5a049bfac0ac05bc8327a38d2c"}
Sep 17 20:51:52.970037 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:52.969913    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7m89j" event={"ID":"102ec935-a1f0-4e2b-84ae-8832e11a1033","Type":"ContainerStarted","Data":"d212be5d0d358d99caac6ba7fd72a241086dd008ff4844255d873121f5367989"}
Sep 17 20:51:53.974704 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:53.974671    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7m89j" event={"ID":"102ec935-a1f0-4e2b-84ae-8832e11a1033","Type":"ContainerStarted","Data":"96b26ef6a2725c04707470b240555bedc967132224b085565a31c09849cf78de"}
Sep 17 20:51:55.982960 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:55.982921    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7m89j" event={"ID":"102ec935-a1f0-4e2b-84ae-8832e11a1033","Type":"ContainerStarted","Data":"b8d095383d51cfbc667adf20fca2084b0722b440a459b5eae17d23f81839d89f"}
Sep 17 20:51:56.006615 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:56.006567    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7m89j" podStartSLOduration=33.853064366 podStartE2EDuration="36.006552603s" totalImagesPullingTime="2.153488237s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:51:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:56.004989166 +0000 UTC m=+157.615722394" watchObservedRunningTime="2026-09-17 20:51:56.006552603 +0000 UTC m=+157.617285811"
Sep 17 20:51:56.126084 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:56.125841    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:51:56.490837 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:56.490806    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-78ncn" podUID="ae0c2874-4c74-491b-8fa4-4ff0fcc84c00"
Sep 17 20:51:56.496915 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:51:56.496895    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-sknnf" podUID="8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7"
Sep 17 20:51:56.985683 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:51:56.985649    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-78ncn"
Sep 17 20:52:01.049811 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.049779    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-brjxp"]
Sep 17 20:52:01.052662 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.052647    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-brjxp"
Sep 17 20:52:01.063675 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.063650    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dac1817a-9871-4cb6-848f-56465179a242-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-brjxp\" (UID: \"dac1817a-9871-4cb6-848f-56465179a242\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-brjxp"
Sep 17 20:52:01.063762 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.063698    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dac1817a-9871-4cb6-848f-56465179a242-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-brjxp\" (UID: \"dac1817a-9871-4cb6-848f-56465179a242\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-brjxp"
Sep 17 20:52:01.063813 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.063751    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dac1817a-9871-4cb6-848f-56465179a242-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-brjxp\" (UID: \"dac1817a-9871-4cb6-848f-56465179a242\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-brjxp"
Sep 17 20:52:01.063813 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.063798    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dac1817a-9871-4cb6-848f-56465179a242-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-brjxp\" (UID: \"dac1817a-9871-4cb6-848f-56465179a242\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-brjxp"
Sep 17 20:52:01.063881 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.063820    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg78q\" (UniqueName: \"kubernetes.io/projected/dac1817a-9871-4cb6-848f-56465179a242-kube-api-access-bg78q\") pod \"kube-state-metrics-84c885658c-brjxp\" (UID: \"dac1817a-9871-4cb6-848f-56465179a242\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-brjxp"
Sep 17 20:52:01.063881 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.063846    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dac1817a-9871-4cb6-848f-56465179a242-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-brjxp\" (UID: \"dac1817a-9871-4cb6-848f-56465179a242\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-brjxp"
Sep 17 20:52:01.065058 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:52:01.065039    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-state-metrics-tls\" is forbidden: User \"system:node:ip-10-0-138-140.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-138-140.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" type="*v1.Secret"
Sep 17 20:52:01.065730 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:52:01.065709    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-state-metrics-custom-resource-state-configmap\" is forbidden: User \"system:node:ip-10-0-138-140.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-138-140.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" type="*v1.ConfigMap"
Sep 17 20:52:01.065850 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.065839    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 20:52:01.066162 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:52:01.066139    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-state-metrics-dockercfg-tjkll\" is forbidden: User \"system:node:ip-10-0-138-140.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-138-140.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-tjkll\"" type="*v1.Secret"
Sep 17 20:52:01.066292 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.066264    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 20:52:01.066496 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.066480    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 20:52:01.067072 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.067057    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 20:52:01.088407 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.088381    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-77pdr"]
Sep 17 20:52:01.091163 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.091144    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-77pdr"
Sep 17 20:52:01.093850 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.093829    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-brjxp"]
Sep 17 20:52:01.094407 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:52:01.094375    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-tls\" is forbidden: User \"system:node:ip-10-0-138-140.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-138-140.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret"
Sep 17 20:52:01.094514 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:52:01.094430    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-dockercfg-jg6qn\" is forbidden: User \"system:node:ip-10-0-138-140.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-138-140.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jg6qn\"" type="*v1.Secret"
Sep 17 20:52:01.096561 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:52:01.096535    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-kube-rbac-proxy-config\" is forbidden: User \"system:node:ip-10-0-138-140.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-138-140.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret"
Sep 17 20:52:01.097022 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.096999    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 20:52:01.164251 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.164223    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e8fa9f0a-d3a0-47b1-943b-3f6c548f4290-node-exporter-wtmp\") pod \"node-exporter-77pdr\" (UID: \"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290\") " pod="openshift-monitoring/node-exporter-77pdr"
Sep 17 20:52:01.164251 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.164254    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e8fa9f0a-d3a0-47b1-943b-3f6c548f4290-root\") pod \"node-exporter-77pdr\" (UID: \"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290\") " pod="openshift-monitoring/node-exporter-77pdr"
Sep 17 20:52:01.164449 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.164270    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8fa9f0a-d3a0-47b1-943b-3f6c548f4290-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-77pdr\" (UID: \"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290\") " pod="openshift-monitoring/node-exporter-77pdr"
Sep 17 20:52:01.164449 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.164292    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg78q\" (UniqueName: \"kubernetes.io/projected/dac1817a-9871-4cb6-848f-56465179a242-kube-api-access-bg78q\") pod \"kube-state-metrics-84c885658c-brjxp\" (UID: \"dac1817a-9871-4cb6-848f-56465179a242\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-brjxp"
Sep 17 20:52:01.164449 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.164323    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e8fa9f0a-d3a0-47b1-943b-3f6c548f4290-node-exporter-accelerators-collector-config\") pod \"node-exporter-77pdr\" (UID: \"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290\") " pod="openshift-monitoring/node-exporter-77pdr"
Sep 17 20:52:01.164449 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.164434    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dac1817a-9871-4cb6-848f-56465179a242-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-brjxp\" (UID: \"dac1817a-9871-4cb6-848f-56465179a242\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-brjxp"
Sep 17 20:52:01.164591 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.164473    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fp85\" (UniqueName: \"kubernetes.io/projected/e8fa9f0a-d3a0-47b1-943b-3f6c548f4290-kube-api-access-5fp85\") pod \"node-exporter-77pdr\" (UID: \"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290\") " pod="openshift-monitoring/node-exporter-77pdr"
Sep 17 20:52:01.164591 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.164505    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8fa9f0a-d3a0-47b1-943b-3f6c548f4290-sys\") pod \"node-exporter-77pdr\" (UID: \"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290\") " pod="openshift-monitoring/node-exporter-77pdr"
Sep 17 20:52:01.164591 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.164526    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e8fa9f0a-d3a0-47b1-943b-3f6c548f4290-node-exporter-textfile\") pod \"node-exporter-77pdr\" (UID: \"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290\") " pod="openshift-monitoring/node-exporter-77pdr"
Sep 17 20:52:01.164591 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.164554    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e8fa9f0a-d3a0-47b1-943b-3f6c548f4290-node-exporter-tls\") pod \"node-exporter-77pdr\" (UID: \"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290\") " pod="openshift-monitoring/node-exporter-77pdr"
Sep 17 20:52:01.164779 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.164659    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dac1817a-9871-4cb6-848f-56465179a242-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-brjxp\" (UID: \"dac1817a-9871-4cb6-848f-56465179a242\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-brjxp"
Sep 17 20:52:01.164779 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.164730    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dac1817a-9871-4cb6-848f-56465179a242-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-brjxp\" (UID: \"dac1817a-9871-4cb6-848f-56465179a242\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-brjxp"
Sep 17 20:52:01.164779 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.164760    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dac1817a-9871-4cb6-848f-56465179a242-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-brjxp\" (UID: \"dac1817a-9871-4cb6-848f-56465179a242\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-brjxp"
Sep 17 20:52:01.164915 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.164786    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8fa9f0a-d3a0-47b1-943b-3f6c548f4290-metrics-client-ca\") pod \"node-exporter-77pdr\" (UID: \"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290\") " pod="openshift-monitoring/node-exporter-77pdr"
Sep 17 20:52:01.164915 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.164836    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dac1817a-9871-4cb6-848f-56465179a242-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-brjxp\" (UID: \"dac1817a-9871-4cb6-848f-56465179a242\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-brjxp"
Sep 17 20:52:01.165215 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.165192    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dac1817a-9871-4cb6-848f-56465179a242-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-brjxp\" (UID: \"dac1817a-9871-4cb6-848f-56465179a242\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-brjxp"
Sep 17 20:52:01.165354 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.165337    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dac1817a-9871-4cb6-848f-56465179a242-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-brjxp\" (UID: \"dac1817a-9871-4cb6-848f-56465179a242\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-brjxp"
Sep 17 20:52:01.167058 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.167033    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dac1817a-9871-4cb6-848f-56465179a242-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-brjxp\" (UID: \"dac1817a-9871-4cb6-848f-56465179a242\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-brjxp"
Sep 17 20:52:01.180800 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.180781    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg78q\" (UniqueName: \"kubernetes.io/projected/dac1817a-9871-4cb6-848f-56465179a242-kube-api-access-bg78q\") pod \"kube-state-metrics-84c885658c-brjxp\" (UID: \"dac1817a-9871-4cb6-848f-56465179a242\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-brjxp"
Sep 17 20:52:01.265193 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.265157    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e8fa9f0a-d3a0-47b1-943b-3f6c548f4290-node-exporter-wtmp\") pod \"node-exporter-77pdr\" (UID: \"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290\") " pod="openshift-monitoring/node-exporter-77pdr"
Sep 17 20:52:01.265193 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.265188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e8fa9f0a-d3a0-47b1-943b-3f6c548f4290-root\") pod \"node-exporter-77pdr\" (UID: \"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290\") " pod="openshift-monitoring/node-exporter-77pdr"
Sep 17 20:52:01.265430 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.265225    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8fa9f0a-d3a0-47b1-943b-3f6c548f4290-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-77pdr\" (UID: \"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290\") " pod="openshift-monitoring/node-exporter-77pdr"
Sep 17 20:52:01.265430 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.265246    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e8fa9f0a-d3a0-47b1-943b-3f6c548f4290-node-exporter-accelerators-collector-config\") pod \"node-exporter-77pdr\" (UID: \"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290\") " pod="openshift-monitoring/node-exporter-77pdr"
Sep 17 20:52:01.265430 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.265274    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fp85\" (UniqueName: \"kubernetes.io/projected/e8fa9f0a-d3a0-47b1-943b-3f6c548f4290-kube-api-access-5fp85\") pod \"node-exporter-77pdr\" (UID: \"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290\") " pod="openshift-monitoring/node-exporter-77pdr"
Sep 17 20:52:01.265430 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.265281    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e8fa9f0a-d3a0-47b1-943b-3f6c548f4290-root\") pod \"node-exporter-77pdr\" (UID: \"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290\") " pod="openshift-monitoring/node-exporter-77pdr"
Sep 17 20:52:01.265430 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.265295    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8fa9f0a-d3a0-47b1-943b-3f6c548f4290-sys\") pod \"node-exporter-77pdr\" (UID: \"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290\") " pod="openshift-monitoring/node-exporter-77pdr"
Sep 17 20:52:01.265430 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.265331    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8fa9f0a-d3a0-47b1-943b-3f6c548f4290-sys\") pod \"node-exporter-77pdr\" (UID: \"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290\") " pod="openshift-monitoring/node-exporter-77pdr"
Sep 17 20:52:01.265430 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.265349    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e8fa9f0a-d3a0-47b1-943b-3f6c548f4290-node-exporter-textfile\") pod \"node-exporter-77pdr\" (UID: \"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290\") " pod="openshift-monitoring/node-exporter-77pdr"
Sep 17 20:52:01.265430 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.265382    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e8fa9f0a-d3a0-47b1-943b-3f6c548f4290-node-exporter-tls\") pod \"node-exporter-77pdr\" (UID: \"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290\") " pod="openshift-monitoring/node-exporter-77pdr"
Sep 17 20:52:01.265430 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.265350    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e8fa9f0a-d3a0-47b1-943b-3f6c548f4290-node-exporter-wtmp\") pod \"node-exporter-77pdr\" (UID: \"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290\") " pod="openshift-monitoring/node-exporter-77pdr"
Sep 17 20:52:01.265699 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.265463    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8fa9f0a-d3a0-47b1-943b-3f6c548f4290-metrics-client-ca\") pod \"node-exporter-77pdr\" (UID: \"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290\") " pod="openshift-monitoring/node-exporter-77pdr"
Sep 17 20:52:01.265699 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.265666    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e8fa9f0a-d3a0-47b1-943b-3f6c548f4290-node-exporter-textfile\") pod \"node-exporter-77pdr\" (UID: \"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290\") " pod="openshift-monitoring/node-exporter-77pdr"
Sep 17 20:52:01.265867 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.265850    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e8fa9f0a-d3a0-47b1-943b-3f6c548f4290-node-exporter-accelerators-collector-config\") pod \"node-exporter-77pdr\" (UID: \"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290\") " pod="openshift-monitoring/node-exporter-77pdr"
Sep 17 20:52:01.265945 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.265930    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8fa9f0a-d3a0-47b1-943b-3f6c548f4290-metrics-client-ca\") pod \"node-exporter-77pdr\" (UID: \"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290\") " pod="openshift-monitoring/node-exporter-77pdr"
Sep 17 20:52:01.276580 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.276533    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fp85\" (UniqueName: \"kubernetes.io/projected/e8fa9f0a-d3a0-47b1-943b-3f6c548f4290-kube-api-access-5fp85\") pod \"node-exporter-77pdr\" (UID: \"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290\") " pod="openshift-monitoring/node-exporter-77pdr"
Sep 17 20:52:01.365821 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.365755    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-metrics-tls\") pod \"dns-default-78ncn\" (UID: \"ae0c2874-4c74-491b-8fa4-4ff0fcc84c00\") " pod="openshift-dns/dns-default-78ncn"
Sep 17 20:52:01.368084 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.368053    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae0c2874-4c74-491b-8fa4-4ff0fcc84c00-metrics-tls\") pod \"dns-default-78ncn\" (UID: \"ae0c2874-4c74-491b-8fa4-4ff0fcc84c00\") " pod="openshift-dns/dns-default-78ncn"
Sep 17 20:52:01.467743 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.467708    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-755b7db98d-9t8hg"]
Sep 17 20:52:01.471632 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.471611    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-755b7db98d-9t8hg"
Sep 17 20:52:01.474092 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.474067    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7mwmf\""
Sep 17 20:52:01.474231 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.474067    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 20:52:01.474231 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.474202    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 20:52:01.474983 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.474965    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 20:52:01.475349 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.475331    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 20:52:01.475437 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.475373    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 20:52:01.475437 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.475411    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 20:52:01.475982 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.475964    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 20:52:01.480118 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.480100    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 20:52:01.484137 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.484116    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-755b7db98d-9t8hg"]
Sep 17 20:52:01.489174 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.489159    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lzdg7\""
Sep 17 20:52:01.497480 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.497460    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-78ncn"
Sep 17 20:52:01.567693 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.567664    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-console-config\") pod \"console-755b7db98d-9t8hg\" (UID: \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\") " pod="openshift-console/console-755b7db98d-9t8hg"
Sep 17 20:52:01.567889 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.567746    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-service-ca\") pod \"console-755b7db98d-9t8hg\" (UID: \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\") " pod="openshift-console/console-755b7db98d-9t8hg"
Sep 17 20:52:01.567889 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.567848    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tkq7\" (UniqueName: \"kubernetes.io/projected/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-kube-api-access-9tkq7\") pod \"console-755b7db98d-9t8hg\" (UID: \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\") " pod="openshift-console/console-755b7db98d-9t8hg"
Sep 17 20:52:01.568025 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.567916    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-console-oauth-config\") pod \"console-755b7db98d-9t8hg\" (UID: \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\") " pod="openshift-console/console-755b7db98d-9t8hg"
Sep 17 20:52:01.568025 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.567989    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-console-serving-cert\") pod \"console-755b7db98d-9t8hg\" (UID: \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\") " pod="openshift-console/console-755b7db98d-9t8hg"
Sep 17 20:52:01.568167 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.568071    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-trusted-ca-bundle\") pod \"console-755b7db98d-9t8hg\" (UID: \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\") " pod="openshift-console/console-755b7db98d-9t8hg"
Sep 17 20:52:01.568167 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.568125    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-oauth-serving-cert\") pod \"console-755b7db98d-9t8hg\" (UID: \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\") " pod="openshift-console/console-755b7db98d-9t8hg"
Sep 17 20:52:01.669427 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.669345    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-trusted-ca-bundle\") pod \"console-755b7db98d-9t8hg\" (UID: \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\") " pod="openshift-console/console-755b7db98d-9t8hg"
Sep 17 20:52:01.669427 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.669377    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-oauth-serving-cert\") pod \"console-755b7db98d-9t8hg\" (UID: \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\") " pod="openshift-console/console-755b7db98d-9t8hg"
Sep 17 20:52:01.669427 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.669413    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-console-config\") pod \"console-755b7db98d-9t8hg\" (UID: \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\") " pod="openshift-console/console-755b7db98d-9t8hg"
Sep 17 20:52:01.669659 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.669523    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-service-ca\") pod \"console-755b7db98d-9t8hg\" (UID: \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\") " pod="openshift-console/console-755b7db98d-9t8hg"
Sep 17 20:52:01.669659 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.669582    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tkq7\" (UniqueName: \"kubernetes.io/projected/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-kube-api-access-9tkq7\") pod \"console-755b7db98d-9t8hg\" (UID: \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\") " pod="openshift-console/console-755b7db98d-9t8hg"
Sep 17 20:52:01.669659 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.669616    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-console-oauth-config\") pod \"console-755b7db98d-9t8hg\" (UID: \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\") " pod="openshift-console/console-755b7db98d-9t8hg"
Sep 17 20:52:01.669659 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.669654    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-console-serving-cert\") pod \"console-755b7db98d-9t8hg\" (UID: \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\") " pod="openshift-console/console-755b7db98d-9t8hg"
Sep 17 20:52:01.670174 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.670153    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-oauth-serving-cert\") pod \"console-755b7db98d-9t8hg\" (UID: \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\") " pod="openshift-console/console-755b7db98d-9t8hg"
Sep 17 20:52:01.670174 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.670169    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-console-config\") pod \"console-755b7db98d-9t8hg\" (UID: \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\") " pod="openshift-console/console-755b7db98d-9t8hg"
Sep 17 20:52:01.670440 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.670417    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-service-ca\") pod \"console-755b7db98d-9t8hg\" (UID: \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\") " pod="openshift-console/console-755b7db98d-9t8hg"
Sep 17 20:52:01.670488 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.670446    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-trusted-ca-bundle\") pod \"console-755b7db98d-9t8hg\" (UID: \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\") " pod="openshift-console/console-755b7db98d-9t8hg"
Sep 17 20:52:01.672067 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.672043    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-console-oauth-config\") pod \"console-755b7db98d-9t8hg\" (UID: \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\") " pod="openshift-console/console-755b7db98d-9t8hg"
Sep 17 20:52:01.672159 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.672144    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-console-serving-cert\") pod \"console-755b7db98d-9t8hg\" (UID: \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\") " pod="openshift-console/console-755b7db98d-9t8hg"
Sep 17 20:52:01.677878 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.677862    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tkq7\" (UniqueName: \"kubernetes.io/projected/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-kube-api-access-9tkq7\") pod \"console-755b7db98d-9t8hg\" (UID: \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\") " pod="openshift-console/console-755b7db98d-9t8hg"
Sep 17 20:52:01.770761 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.770736    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-78ncn"]
Sep 17 20:52:01.773112 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:52:01.773089    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae0c2874_4c74_491b_8fa4_4ff0fcc84c00.slice/crio-6a0c7eec774570041c412dac55ed7d47ec8c028e607adb1c683d3ee714c6d2d7 WatchSource:0}: Error finding container 6a0c7eec774570041c412dac55ed7d47ec8c028e607adb1c683d3ee714c6d2d7: Status 404 returned error can't find the container with id 6a0c7eec774570041c412dac55ed7d47ec8c028e607adb1c683d3ee714c6d2d7
Sep 17 20:52:01.781016 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.780992    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-755b7db98d-9t8hg"
Sep 17 20:52:01.917354 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.917326    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 20:52:01.927937 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.927890    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8fa9f0a-d3a0-47b1-943b-3f6c548f4290-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-77pdr\" (UID: \"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290\") " pod="openshift-monitoring/node-exporter-77pdr"
Sep 17 20:52:01.967198 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.967171    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jg6qn\""
Sep 17 20:52:01.970679 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.970664    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 20:52:01.975259 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:01.975231    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dac1817a-9871-4cb6-848f-56465179a242-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-brjxp\" (UID: \"dac1817a-9871-4cb6-848f-56465179a242\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-brjxp"
Sep 17 20:52:02.001293 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.001257    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-78ncn" event={"ID":"ae0c2874-4c74-491b-8fa4-4ff0fcc84c00","Type":"ContainerStarted","Data":"6a0c7eec774570041c412dac55ed7d47ec8c028e607adb1c683d3ee714c6d2d7"}
Sep 17 20:52:02.050857 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.050837    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-755b7db98d-9t8hg"]
Sep 17 20:52:02.053005 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:52:02.052985    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57891b1c_80f0_489b_a6d6_c0ca2287ccb2.slice/crio-a1ef5edfd4ddb04f403cb63cf410cdd13dba4d0b19730283eb360e7c623322ca WatchSource:0}: Error finding container a1ef5edfd4ddb04f403cb63cf410cdd13dba4d0b19730283eb360e7c623322ca: Status 404 returned error can't find the container with id a1ef5edfd4ddb04f403cb63cf410cdd13dba4d0b19730283eb360e7c623322ca
Sep 17 20:52:02.138103 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.137866    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 20:52:02.144891 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.144870    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.152765 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.152200    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 20:52:02.152765 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.152453    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-58dfl\""
Sep 17 20:52:02.152765 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.152559    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 20:52:02.152765 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.152572    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 20:52:02.152765 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.152588    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 20:52:02.152765 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.152660    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 20:52:02.152765 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.152715    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 20:52:02.153160 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.152881    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 20:52:02.153160 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.152923    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 20:52:02.153160 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.153019    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 20:52:02.165032 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:52:02.165018    2566 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: failed to sync secret cache: timed out waiting for the condition
Sep 17 20:52:02.165134 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:52:02.165095    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dac1817a-9871-4cb6-848f-56465179a242-kube-state-metrics-tls podName:dac1817a-9871-4cb6-848f-56465179a242 nodeName:}" failed. No retries permitted until 2026-09-17 20:52:02.665074931 +0000 UTC m=+164.275808158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/dac1817a-9871-4cb6-848f-56465179a242-kube-state-metrics-tls") pod "kube-state-metrics-84c885658c-brjxp" (UID: "dac1817a-9871-4cb6-848f-56465179a242") : failed to sync secret cache: timed out waiting for the condition
Sep 17 20:52:02.174623 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.174596    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/91ea8785-e65b-4c0c-bbb7-e5514d627965-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.174721 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.174641    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91ea8785-e65b-4c0c-bbb7-e5514d627965-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.174721 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.174673    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/91ea8785-e65b-4c0c-bbb7-e5514d627965-config-out\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.174830 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.174718    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91ea8785-e65b-4c0c-bbb7-e5514d627965-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.174887 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.174838    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lphvc\" (UniqueName: \"kubernetes.io/projected/91ea8785-e65b-4c0c-bbb7-e5514d627965-kube-api-access-lphvc\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.174938 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.174890    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.174989 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.174933    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.175046 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.175007    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.175046 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.175040    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/91ea8785-e65b-4c0c-bbb7-e5514d627965-tls-assets\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.175139 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.175062    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.175139 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.175085    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-config-volume\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.175139 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.175115    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-web-config\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.175312 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.175139    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.175312 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.175158    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 20:52:02.210900 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.210847    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 20:52:02.265886 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:52:02.265853    2566 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: failed to sync secret cache: timed out waiting for the condition
Sep 17 20:52:02.266006 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:52:02.265940    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8fa9f0a-d3a0-47b1-943b-3f6c548f4290-node-exporter-tls podName:e8fa9f0a-d3a0-47b1-943b-3f6c548f4290 nodeName:}" failed. No retries permitted until 2026-09-17 20:52:02.765919235 +0000 UTC m=+164.376652442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/e8fa9f0a-d3a0-47b1-943b-3f6c548f4290-node-exporter-tls") pod "node-exporter-77pdr" (UID: "e8fa9f0a-d3a0-47b1-943b-3f6c548f4290") : failed to sync secret cache: timed out waiting for the condition
Sep 17 20:52:02.276446 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.276412    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lphvc\" (UniqueName: \"kubernetes.io/projected/91ea8785-e65b-4c0c-bbb7-e5514d627965-kube-api-access-lphvc\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.276554 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.276468    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.276554 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.276507    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.276671 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.276602    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.276671 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.276634    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/91ea8785-e65b-4c0c-bbb7-e5514d627965-tls-assets\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.276671 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.276662    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.276818 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.276685    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-config-volume\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.276818 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.276718    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-web-config\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.276818 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.276743    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.277379 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.277359    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/91ea8785-e65b-4c0c-bbb7-e5514d627965-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.277504 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.277395    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91ea8785-e65b-4c0c-bbb7-e5514d627965-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.277504 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.277423    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/91ea8785-e65b-4c0c-bbb7-e5514d627965-config-out\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.277504 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.277445    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91ea8785-e65b-4c0c-bbb7-e5514d627965-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.277973 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:52:02.277943    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91ea8785-e65b-4c0c-bbb7-e5514d627965-alertmanager-trusted-ca-bundle podName:91ea8785-e65b-4c0c-bbb7-e5514d627965 nodeName:}" failed. No retries permitted until 2026-09-17 20:52:02.777908887 +0000 UTC m=+164.388642094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/91ea8785-e65b-4c0c-bbb7-e5514d627965-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "91ea8785-e65b-4c0c-bbb7-e5514d627965") : configmap references non-existent config key: ca-bundle.crt
Sep 17 20:52:02.279005 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.278982    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91ea8785-e65b-4c0c-bbb7-e5514d627965-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.279583 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.279482    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/91ea8785-e65b-4c0c-bbb7-e5514d627965-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.280460 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.280436    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.280873 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.280853    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.282318 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.282293    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.282409 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.282303    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-web-config\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.282806 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.282784    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-config-volume\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.282957 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.282924    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.282957 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.282845    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/91ea8785-e65b-4c0c-bbb7-e5514d627965-config-out\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.283378 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.283354    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/91ea8785-e65b-4c0c-bbb7-e5514d627965-tls-assets\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.283473 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.283381    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.285797 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.285775    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lphvc\" (UniqueName: \"kubernetes.io/projected/91ea8785-e65b-4c0c-bbb7-e5514d627965-kube-api-access-lphvc\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.621305 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.621153    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-tjkll\""
Sep 17 20:52:02.681304 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.680666    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dac1817a-9871-4cb6-848f-56465179a242-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-brjxp\" (UID: \"dac1817a-9871-4cb6-848f-56465179a242\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-brjxp"
Sep 17 20:52:02.691978 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.688569    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dac1817a-9871-4cb6-848f-56465179a242-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-brjxp\" (UID: \"dac1817a-9871-4cb6-848f-56465179a242\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-brjxp"
Sep 17 20:52:02.691978 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.691076    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 20:52:02.782052 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.782015    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e8fa9f0a-d3a0-47b1-943b-3f6c548f4290-node-exporter-tls\") pod \"node-exporter-77pdr\" (UID: \"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290\") " pod="openshift-monitoring/node-exporter-77pdr"
Sep 17 20:52:02.782261 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.782162    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91ea8785-e65b-4c0c-bbb7-e5514d627965-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.783417 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.783366    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91ea8785-e65b-4c0c-bbb7-e5514d627965-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:02.785856 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.785830    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e8fa9f0a-d3a0-47b1-943b-3f6c548f4290-node-exporter-tls\") pod \"node-exporter-77pdr\" (UID: \"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290\") " pod="openshift-monitoring/node-exporter-77pdr"
Sep 17 20:52:02.861404 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.861360    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-brjxp"
Sep 17 20:52:02.900833 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:02.900684    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-77pdr"
Sep 17 20:52:03.008304 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:03.008263    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-755b7db98d-9t8hg" event={"ID":"57891b1c-80f0-489b-a6d6-c0ca2287ccb2","Type":"ContainerStarted","Data":"a1ef5edfd4ddb04f403cb63cf410cdd13dba4d0b19730283eb360e7c623322ca"}
Sep 17 20:52:03.054793 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:03.054761    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:52:03.175280 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:03.175129    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-brjxp"]
Sep 17 20:52:03.582868 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:52:03.582837    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac1817a_9871_4cb6_848f_56465179a242.slice/crio-c845925a38471b0b2bb62423cf927e407272c95f4827ddefc3134f51a8837c2b WatchSource:0}: Error finding container c845925a38471b0b2bb62423cf927e407272c95f4827ddefc3134f51a8837c2b: Status 404 returned error can't find the container with id c845925a38471b0b2bb62423cf927e407272c95f4827ddefc3134f51a8837c2b
Sep 17 20:52:04.014416 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:04.014366    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-brjxp" event={"ID":"dac1817a-9871-4cb6-848f-56465179a242","Type":"ContainerStarted","Data":"c845925a38471b0b2bb62423cf927e407272c95f4827ddefc3134f51a8837c2b"}
Sep 17 20:52:06.237639 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:52:06.237607    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8fa9f0a_d3a0_47b1_943b_3f6c548f4290.slice/crio-6b14dc79228dff77057cd3545abcee416d498cff607f034a697fca904f375fbe WatchSource:0}: Error finding container 6b14dc79228dff77057cd3545abcee416d498cff607f034a697fca904f375fbe: Status 404 returned error can't find the container with id 6b14dc79228dff77057cd3545abcee416d498cff607f034a697fca904f375fbe
Sep 17 20:52:06.519926 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:06.519896    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 20:52:06.612201 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:52:06.612166    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ea8785_e65b_4c0c_bbb7_e5514d627965.slice/crio-be5da4ac3ad701fe8b2c9ec4a1d040755da52981e58ac54eb23eaf89855df38a WatchSource:0}: Error finding container be5da4ac3ad701fe8b2c9ec4a1d040755da52981e58ac54eb23eaf89855df38a: Status 404 returned error can't find the container with id be5da4ac3ad701fe8b2c9ec4a1d040755da52981e58ac54eb23eaf89855df38a
Sep 17 20:52:07.030628 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.030048    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-78ncn" event={"ID":"ae0c2874-4c74-491b-8fa4-4ff0fcc84c00","Type":"ContainerStarted","Data":"67bd400a5c30f437a30f80de79530850068d9789557a85654c651b6255fb8160"}
Sep 17 20:52:07.030628 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.030109    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-78ncn" event={"ID":"ae0c2874-4c74-491b-8fa4-4ff0fcc84c00","Type":"ContainerStarted","Data":"81ebf460214b39ffef62d54519e3043179afe193e20d5256a6e97687c0c48426"}
Sep 17 20:52:07.030628 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.030186    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-78ncn"
Sep 17 20:52:07.032746 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.032715    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-77pdr" event={"ID":"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290","Type":"ContainerStarted","Data":"6b14dc79228dff77057cd3545abcee416d498cff607f034a697fca904f375fbe"}
Sep 17 20:52:07.035224 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.035184    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-755b7db98d-9t8hg" event={"ID":"57891b1c-80f0-489b-a6d6-c0ca2287ccb2","Type":"ContainerStarted","Data":"949a734ddf74f21af4aabd097bde724d36522c7b1660337764ecb66402373789"}
Sep 17 20:52:07.038298 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.038272    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-brjxp" event={"ID":"dac1817a-9871-4cb6-848f-56465179a242","Type":"ContainerStarted","Data":"227c45b1d15a44b3ef10955e193ca231375775b5567acc5a2aa8e91bfd357afd"}
Sep 17 20:52:07.038423 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.038325    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-brjxp" event={"ID":"dac1817a-9871-4cb6-848f-56465179a242","Type":"ContainerStarted","Data":"821cac3ce1bdc5b347b569746f21e078f59fd3c192e2a66d64f4c19d9402561d"}
Sep 17 20:52:07.038423 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.038354    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-brjxp" event={"ID":"dac1817a-9871-4cb6-848f-56465179a242","Type":"ContainerStarted","Data":"79d15fac2c3b97855ebce6a3ed8d197b22384c149d3ffcd32008a8d0ccabfe9c"}
Sep 17 20:52:07.040470 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.040387    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91ea8785-e65b-4c0c-bbb7-e5514d627965","Type":"ContainerStarted","Data":"be5da4ac3ad701fe8b2c9ec4a1d040755da52981e58ac54eb23eaf89855df38a"}
Sep 17 20:52:07.050896 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.050084    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-78ncn" podStartSLOduration=129.589507309 podStartE2EDuration="2m14.050065812s" totalImagesPullingTime="4.460558503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:52:07.048147382 +0000 UTC m=+168.658880634" watchObservedRunningTime="2026-09-17 20:52:07.050065812 +0000 UTC m=+168.660799041"
Sep 17 20:52:07.069912 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.069866    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-brjxp" podStartSLOduration=2.993180495 podStartE2EDuration="6.06984739s" totalImagesPullingTime="3.076666895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:52:07.069128316 +0000 UTC m=+168.679861553" watchObservedRunningTime="2026-09-17 20:52:07.06984739 +0000 UTC m=+168.680580620"
Sep 17 20:52:07.087883 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.087537    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-755b7db98d-9t8hg" podStartSLOduration=1.4833011329999999 podStartE2EDuration="6.087526497s" totalImagesPullingTime="4.604225364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:52:07.087344536 +0000 UTC m=+168.698077767" watchObservedRunningTime="2026-09-17 20:52:07.087526497 +0000 UTC m=+168.698259728"
Sep 17 20:52:07.272651 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.272619    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-755b7db98d-9t8hg"]
Sep 17 20:52:07.308705 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.308634    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66c8bb9b49-bd9vr"]
Sep 17 20:52:07.312225 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.312173    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66c8bb9b49-bd9vr"
Sep 17 20:52:07.326059 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.326035    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66c8bb9b49-bd9vr"]
Sep 17 20:52:07.424133 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.424103    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/34006e50-f738-41b0-b6e4-055e6694bad5-console-config\") pod \"console-66c8bb9b49-bd9vr\" (UID: \"34006e50-f738-41b0-b6e4-055e6694bad5\") " pod="openshift-console/console-66c8bb9b49-bd9vr"
Sep 17 20:52:07.424323 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.424142    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34006e50-f738-41b0-b6e4-055e6694bad5-oauth-serving-cert\") pod \"console-66c8bb9b49-bd9vr\" (UID: \"34006e50-f738-41b0-b6e4-055e6694bad5\") " pod="openshift-console/console-66c8bb9b49-bd9vr"
Sep 17 20:52:07.424323 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.424186    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34006e50-f738-41b0-b6e4-055e6694bad5-console-serving-cert\") pod \"console-66c8bb9b49-bd9vr\" (UID: \"34006e50-f738-41b0-b6e4-055e6694bad5\") " pod="openshift-console/console-66c8bb9b49-bd9vr"
Sep 17 20:52:07.424323 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.424237    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/34006e50-f738-41b0-b6e4-055e6694bad5-service-ca\") pod \"console-66c8bb9b49-bd9vr\" (UID: \"34006e50-f738-41b0-b6e4-055e6694bad5\") " pod="openshift-console/console-66c8bb9b49-bd9vr"
Sep 17 20:52:07.424323 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.424265    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34006e50-f738-41b0-b6e4-055e6694bad5-console-oauth-config\") pod \"console-66c8bb9b49-bd9vr\" (UID: \"34006e50-f738-41b0-b6e4-055e6694bad5\") " pod="openshift-console/console-66c8bb9b49-bd9vr"
Sep 17 20:52:07.424323 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.424290    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34006e50-f738-41b0-b6e4-055e6694bad5-trusted-ca-bundle\") pod \"console-66c8bb9b49-bd9vr\" (UID: \"34006e50-f738-41b0-b6e4-055e6694bad5\") " pod="openshift-console/console-66c8bb9b49-bd9vr"
Sep 17 20:52:07.424515 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.424333    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8tl5\" (UniqueName: \"kubernetes.io/projected/34006e50-f738-41b0-b6e4-055e6694bad5-kube-api-access-b8tl5\") pod \"console-66c8bb9b49-bd9vr\" (UID: \"34006e50-f738-41b0-b6e4-055e6694bad5\") " pod="openshift-console/console-66c8bb9b49-bd9vr"
Sep 17 20:52:07.524705 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.524670    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34006e50-f738-41b0-b6e4-055e6694bad5-console-serving-cert\") pod \"console-66c8bb9b49-bd9vr\" (UID: \"34006e50-f738-41b0-b6e4-055e6694bad5\") " pod="openshift-console/console-66c8bb9b49-bd9vr"
Sep 17 20:52:07.524705 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.524705    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/34006e50-f738-41b0-b6e4-055e6694bad5-service-ca\") pod \"console-66c8bb9b49-bd9vr\" (UID: \"34006e50-f738-41b0-b6e4-055e6694bad5\") " pod="openshift-console/console-66c8bb9b49-bd9vr"
Sep 17 20:52:07.524937 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.524723    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34006e50-f738-41b0-b6e4-055e6694bad5-console-oauth-config\") pod \"console-66c8bb9b49-bd9vr\" (UID: \"34006e50-f738-41b0-b6e4-055e6694bad5\") " pod="openshift-console/console-66c8bb9b49-bd9vr"
Sep 17 20:52:07.524937 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.524753    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34006e50-f738-41b0-b6e4-055e6694bad5-trusted-ca-bundle\") pod \"console-66c8bb9b49-bd9vr\" (UID: \"34006e50-f738-41b0-b6e4-055e6694bad5\") " pod="openshift-console/console-66c8bb9b49-bd9vr"
Sep 17 20:52:07.525048 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.524942    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8tl5\" (UniqueName: \"kubernetes.io/projected/34006e50-f738-41b0-b6e4-055e6694bad5-kube-api-access-b8tl5\") pod \"console-66c8bb9b49-bd9vr\" (UID: \"34006e50-f738-41b0-b6e4-055e6694bad5\") " pod="openshift-console/console-66c8bb9b49-bd9vr"
Sep 17 20:52:07.525117 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.525095    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/34006e50-f738-41b0-b6e4-055e6694bad5-console-config\") pod \"console-66c8bb9b49-bd9vr\" (UID: \"34006e50-f738-41b0-b6e4-055e6694bad5\") " pod="openshift-console/console-66c8bb9b49-bd9vr"
Sep 17 20:52:07.525474 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.525179    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34006e50-f738-41b0-b6e4-055e6694bad5-oauth-serving-cert\") pod \"console-66c8bb9b49-bd9vr\" (UID: \"34006e50-f738-41b0-b6e4-055e6694bad5\") " pod="openshift-console/console-66c8bb9b49-bd9vr"
Sep 17 20:52:07.525474 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.525412    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/34006e50-f738-41b0-b6e4-055e6694bad5-service-ca\") pod \"console-66c8bb9b49-bd9vr\" (UID: \"34006e50-f738-41b0-b6e4-055e6694bad5\") " pod="openshift-console/console-66c8bb9b49-bd9vr"
Sep 17 20:52:07.525757 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.525574    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34006e50-f738-41b0-b6e4-055e6694bad5-trusted-ca-bundle\") pod \"console-66c8bb9b49-bd9vr\" (UID: \"34006e50-f738-41b0-b6e4-055e6694bad5\") " pod="openshift-console/console-66c8bb9b49-bd9vr"
Sep 17 20:52:07.525835 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.525766    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34006e50-f738-41b0-b6e4-055e6694bad5-oauth-serving-cert\") pod \"console-66c8bb9b49-bd9vr\" (UID: \"34006e50-f738-41b0-b6e4-055e6694bad5\") " pod="openshift-console/console-66c8bb9b49-bd9vr"
Sep 17 20:52:07.526125 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.526103    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/34006e50-f738-41b0-b6e4-055e6694bad5-console-config\") pod \"console-66c8bb9b49-bd9vr\" (UID: \"34006e50-f738-41b0-b6e4-055e6694bad5\") " pod="openshift-console/console-66c8bb9b49-bd9vr"
Sep 17 20:52:07.527599 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.527579    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34006e50-f738-41b0-b6e4-055e6694bad5-console-oauth-config\") pod \"console-66c8bb9b49-bd9vr\" (UID: \"34006e50-f738-41b0-b6e4-055e6694bad5\") " pod="openshift-console/console-66c8bb9b49-bd9vr"
Sep 17 20:52:07.527812 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.527793    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34006e50-f738-41b0-b6e4-055e6694bad5-console-serving-cert\") pod \"console-66c8bb9b49-bd9vr\" (UID: \"34006e50-f738-41b0-b6e4-055e6694bad5\") " pod="openshift-console/console-66c8bb9b49-bd9vr"
Sep 17 20:52:07.533910 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.533889    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8tl5\" (UniqueName: \"kubernetes.io/projected/34006e50-f738-41b0-b6e4-055e6694bad5-kube-api-access-b8tl5\") pod \"console-66c8bb9b49-bd9vr\" (UID: \"34006e50-f738-41b0-b6e4-055e6694bad5\") " pod="openshift-console/console-66c8bb9b49-bd9vr"
Sep 17 20:52:07.621985 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.621959    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66c8bb9b49-bd9vr"
Sep 17 20:52:07.899031 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:07.898953    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66c8bb9b49-bd9vr"]
Sep 17 20:52:07.901899 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:52:07.901873    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34006e50_f738_41b0_b6e4_055e6694bad5.slice/crio-8c720dc08057a06d0c35e25b8ee756119ea32c222642150cc62671d35a24508d WatchSource:0}: Error finding container 8c720dc08057a06d0c35e25b8ee756119ea32c222642150cc62671d35a24508d: Status 404 returned error can't find the container with id 8c720dc08057a06d0c35e25b8ee756119ea32c222642150cc62671d35a24508d
Sep 17 20:52:08.045980 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:08.045949    2566 generic.go:356] "Generic (PLEG): container finished" podID="e8fa9f0a-d3a0-47b1-943b-3f6c548f4290" containerID="4dc18413261fa62d009ea434bbcce9d58d2d3e2337b2cade5a53d6246fc9fa28" exitCode=0
Sep 17 20:52:08.046165 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:08.046023    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-77pdr" event={"ID":"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290","Type":"ContainerDied","Data":"4dc18413261fa62d009ea434bbcce9d58d2d3e2337b2cade5a53d6246fc9fa28"}
Sep 17 20:52:08.047821 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:08.047799    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66c8bb9b49-bd9vr" event={"ID":"34006e50-f738-41b0-b6e4-055e6694bad5","Type":"ContainerStarted","Data":"83cfb166d9944df45d1a8da44cf4f0f14b751af0d1e802897e8bded31a9f1664"}
Sep 17 20:52:08.047821 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:08.047834    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66c8bb9b49-bd9vr" event={"ID":"34006e50-f738-41b0-b6e4-055e6694bad5","Type":"ContainerStarted","Data":"8c720dc08057a06d0c35e25b8ee756119ea32c222642150cc62671d35a24508d"}
Sep 17 20:52:08.049721 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:08.049510    2566 generic.go:356] "Generic (PLEG): container finished" podID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerID="7c2373ad7c629feb54a23ba725230135fd1123b803aa16c858714713b1d0d5cd" exitCode=0
Sep 17 20:52:08.049721 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:08.049552    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91ea8785-e65b-4c0c-bbb7-e5514d627965","Type":"ContainerDied","Data":"7c2373ad7c629feb54a23ba725230135fd1123b803aa16c858714713b1d0d5cd"}
Sep 17 20:52:08.096672 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:08.096630    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66c8bb9b49-bd9vr" podStartSLOduration=1.096615988 podStartE2EDuration="1.096615988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:52:08.094628992 +0000 UTC m=+169.705362222" watchObservedRunningTime="2026-09-17 20:52:08.096615988 +0000 UTC m=+169.707349216"
Sep 17 20:52:09.056072 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:09.056033    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-77pdr" event={"ID":"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290","Type":"ContainerStarted","Data":"037b4989e68e90dd35e5e6dc6937ab38fb18560d66a8153dec7331cccd7fffc9"}
Sep 17 20:52:09.056530 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:09.056101    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-77pdr" event={"ID":"e8fa9f0a-d3a0-47b1-943b-3f6c548f4290","Type":"ContainerStarted","Data":"e9372457b7522c9f58c1f4dbe775cd60f86029236d850a876308752a1367217e"}
Sep 17 20:52:09.079581 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:09.079534    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-77pdr" podStartSLOduration=6.708749222 podStartE2EDuration="8.079517945s" totalImagesPullingTime="1.299036229s" totalInitContainerRuntime="71.732494ms" isStatefulPod=true podCreationTimestamp="2026-09-17 20:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:52:09.076863734 +0000 UTC m=+170.687596964" watchObservedRunningTime="2026-09-17 20:52:09.079517945 +0000 UTC m=+170.690251174"
Sep 17 20:52:10.064191 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:10.064154    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91ea8785-e65b-4c0c-bbb7-e5514d627965","Type":"ContainerStarted","Data":"7db8b1255759bd301781b7f87222a4ebef786447d6320d6c22c98b3d3501f3f0"}
Sep 17 20:52:10.064598 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:10.064278    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91ea8785-e65b-4c0c-bbb7-e5514d627965","Type":"ContainerStarted","Data":"c5eae3cd250ec50d6d607ddf7eda3b7d78271cc2f0c62c2c6e956a45a5b80ad8"}
Sep 17 20:52:10.064598 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:10.064324    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91ea8785-e65b-4c0c-bbb7-e5514d627965","Type":"ContainerStarted","Data":"ed264130222cf71c748308eab5a5bfea546aca339e0b3df8fb5806ce8af79beb"}
Sep 17 20:52:10.064598 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:10.064363    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91ea8785-e65b-4c0c-bbb7-e5514d627965","Type":"ContainerStarted","Data":"7d204c09ef813ff3b720df3d5757984346d9655189a033dfb47f417f65a2fb79"}
Sep 17 20:52:10.064598 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:10.064409    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91ea8785-e65b-4c0c-bbb7-e5514d627965","Type":"ContainerStarted","Data":"5f1fbb1b6581521647a2c27e05afcaa1a34b9458dd7de9f6e826b78fd8ce1f47"}
Sep 17 20:52:10.456148 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:10.456121    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:52:11.071278 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:11.071246    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91ea8785-e65b-4c0c-bbb7-e5514d627965","Type":"ContainerStarted","Data":"731b49cb3d2244665f6c31aee09ba637cd096d32f8924c7189afd0de00118ce7"}
Sep 17 20:52:11.098008 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:11.097958    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=6.141083523 podStartE2EDuration="9.097944653s" totalImagesPullingTime="2.940127694s" totalInitContainerRuntime="16.733436ms" isStatefulPod=false podCreationTimestamp="2026-09-17 20:52:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:52:11.096424815 +0000 UTC m=+172.707158068" watchObservedRunningTime="2026-09-17 20:52:11.097944653 +0000 UTC m=+172.708677882"
Sep 17 20:52:11.138606 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:11.138570    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg" podUID="1552a7fa-597f-406a-8c77-cca66297356d" containerName="registry" containerID="cri-o://ea8b815715f63c2be65ab3b71e0c10b3c840085e7d71e3f22e7959b3536be19c" gracePeriod=30
Sep 17 20:52:11.378924 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:11.378896    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:52:11.459622 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:11.459594    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1552a7fa-597f-406a-8c77-cca66297356d-image-registry-private-configuration\") pod \"1552a7fa-597f-406a-8c77-cca66297356d\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") "
Sep 17 20:52:11.459794 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:11.459630    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-bound-sa-token\") pod \"1552a7fa-597f-406a-8c77-cca66297356d\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") "
Sep 17 20:52:11.459794 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:11.459657    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1552a7fa-597f-406a-8c77-cca66297356d-installation-pull-secrets\") pod \"1552a7fa-597f-406a-8c77-cca66297356d\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") "
Sep 17 20:52:11.459794 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:11.459690    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1552a7fa-597f-406a-8c77-cca66297356d-registry-certificates\") pod \"1552a7fa-597f-406a-8c77-cca66297356d\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") "
Sep 17 20:52:11.459794 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:11.459730    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-registry-tls\") pod \"1552a7fa-597f-406a-8c77-cca66297356d\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") "
Sep 17 20:52:11.459794 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:11.459761    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1552a7fa-597f-406a-8c77-cca66297356d-ca-trust-extracted\") pod \"1552a7fa-597f-406a-8c77-cca66297356d\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") "
Sep 17 20:52:11.459794 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:11.459786    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1552a7fa-597f-406a-8c77-cca66297356d-trusted-ca\") pod \"1552a7fa-597f-406a-8c77-cca66297356d\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") "
Sep 17 20:52:11.460082 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:11.459823    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkkcr\" (UniqueName: \"kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-kube-api-access-nkkcr\") pod \"1552a7fa-597f-406a-8c77-cca66297356d\" (UID: \"1552a7fa-597f-406a-8c77-cca66297356d\") "
Sep 17 20:52:11.460405 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:11.460236    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1552a7fa-597f-406a-8c77-cca66297356d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1552a7fa-597f-406a-8c77-cca66297356d" (UID: "1552a7fa-597f-406a-8c77-cca66297356d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:52:11.460517 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:11.460459    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1552a7fa-597f-406a-8c77-cca66297356d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1552a7fa-597f-406a-8c77-cca66297356d" (UID: "1552a7fa-597f-406a-8c77-cca66297356d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:52:11.462126 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:11.462098    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1552a7fa-597f-406a-8c77-cca66297356d" (UID: "1552a7fa-597f-406a-8c77-cca66297356d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:52:11.462326 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:11.462302    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1552a7fa-597f-406a-8c77-cca66297356d" (UID: "1552a7fa-597f-406a-8c77-cca66297356d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:52:11.462326 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:11.462310    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1552a7fa-597f-406a-8c77-cca66297356d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1552a7fa-597f-406a-8c77-cca66297356d" (UID: "1552a7fa-597f-406a-8c77-cca66297356d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:52:11.462627 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:11.462609    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1552a7fa-597f-406a-8c77-cca66297356d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1552a7fa-597f-406a-8c77-cca66297356d" (UID: "1552a7fa-597f-406a-8c77-cca66297356d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:52:11.462684 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:11.462622    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-kube-api-access-nkkcr" (OuterVolumeSpecName: "kube-api-access-nkkcr") pod "1552a7fa-597f-406a-8c77-cca66297356d" (UID: "1552a7fa-597f-406a-8c77-cca66297356d"). InnerVolumeSpecName "kube-api-access-nkkcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:52:11.468017 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:11.467997    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1552a7fa-597f-406a-8c77-cca66297356d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1552a7fa-597f-406a-8c77-cca66297356d" (UID: "1552a7fa-597f-406a-8c77-cca66297356d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:52:11.560939 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:11.560907    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1552a7fa-597f-406a-8c77-cca66297356d-trusted-ca\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:52:11.560939 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:11.560933    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkkcr\" (UniqueName: \"kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-kube-api-access-nkkcr\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:52:11.561105 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:11.560950    2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1552a7fa-597f-406a-8c77-cca66297356d-image-registry-private-configuration\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:52:11.561105 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:11.560962    2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-bound-sa-token\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:52:11.561105 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:11.560976    2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1552a7fa-597f-406a-8c77-cca66297356d-installation-pull-secrets\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:52:11.561105 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:11.560987    2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1552a7fa-597f-406a-8c77-cca66297356d-registry-certificates\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:52:11.561105 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:11.560998    2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1552a7fa-597f-406a-8c77-cca66297356d-registry-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:52:11.561105 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:11.561010    2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1552a7fa-597f-406a-8c77-cca66297356d-ca-trust-extracted\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:52:11.781771 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:11.781739    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-755b7db98d-9t8hg"
Sep 17 20:52:12.076548 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:12.076458    2566 generic.go:356] "Generic (PLEG): container finished" podID="1552a7fa-597f-406a-8c77-cca66297356d" containerID="ea8b815715f63c2be65ab3b71e0c10b3c840085e7d71e3f22e7959b3536be19c" exitCode=0
Sep 17 20:52:12.076548 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:12.076507    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg"
Sep 17 20:52:12.077042 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:12.076550    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg" event={"ID":"1552a7fa-597f-406a-8c77-cca66297356d","Type":"ContainerDied","Data":"ea8b815715f63c2be65ab3b71e0c10b3c840085e7d71e3f22e7959b3536be19c"}
Sep 17 20:52:12.077042 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:12.076604    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d97bd56b7-ghblg" event={"ID":"1552a7fa-597f-406a-8c77-cca66297356d","Type":"ContainerDied","Data":"9897121a05cbdc545367d7de09b8a582f7cb1829eb92bc69c6c86bf6283c0215"}
Sep 17 20:52:12.077042 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:12.076634    2566 scope.go:117] "RemoveContainer" containerID="ea8b815715f63c2be65ab3b71e0c10b3c840085e7d71e3f22e7959b3536be19c"
Sep 17 20:52:12.084900 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:12.084884    2566 scope.go:117] "RemoveContainer" containerID="ea8b815715f63c2be65ab3b71e0c10b3c840085e7d71e3f22e7959b3536be19c"
Sep 17 20:52:12.085142 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:52:12.085126    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea8b815715f63c2be65ab3b71e0c10b3c840085e7d71e3f22e7959b3536be19c\": container with ID starting with ea8b815715f63c2be65ab3b71e0c10b3c840085e7d71e3f22e7959b3536be19c not found: ID does not exist" containerID="ea8b815715f63c2be65ab3b71e0c10b3c840085e7d71e3f22e7959b3536be19c"
Sep 17 20:52:12.085179 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:12.085149    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea8b815715f63c2be65ab3b71e0c10b3c840085e7d71e3f22e7959b3536be19c"} err="failed to get container status \"ea8b815715f63c2be65ab3b71e0c10b3c840085e7d71e3f22e7959b3536be19c\": rpc error: code = NotFound desc = could not find container \"ea8b815715f63c2be65ab3b71e0c10b3c840085e7d71e3f22e7959b3536be19c\": container with ID starting with ea8b815715f63c2be65ab3b71e0c10b3c840085e7d71e3f22e7959b3536be19c not found: ID does not exist"
Sep 17 20:52:12.097568 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:12.097545    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d97bd56b7-ghblg"]
Sep 17 20:52:12.101500 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:12.101481    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6d97bd56b7-ghblg"]
Sep 17 20:52:12.459265 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:12.459185    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1552a7fa-597f-406a-8c77-cca66297356d" path="/var/lib/kubelet/pods/1552a7fa-597f-406a-8c77-cca66297356d/volumes"
Sep 17 20:52:17.052140 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:17.052111    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-78ncn"
Sep 17 20:52:17.623003 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:17.622965    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66c8bb9b49-bd9vr"
Sep 17 20:52:17.623151 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:17.623026    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66c8bb9b49-bd9vr"
Sep 17 20:52:17.627760 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:17.627739    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66c8bb9b49-bd9vr"
Sep 17 20:52:18.104318 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:18.104289    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66c8bb9b49-bd9vr"
Sep 17 20:52:34.076522 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:34.076463    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-755b7db98d-9t8hg" podUID="57891b1c-80f0-489b-a6d6-c0ca2287ccb2" containerName="console" containerID="cri-o://949a734ddf74f21af4aabd097bde724d36522c7b1660337764ecb66402373789" gracePeriod=15
Sep 17 20:52:34.305632 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:34.305610    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-755b7db98d-9t8hg"
Sep 17 20:52:34.443747 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:34.443725    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-oauth-serving-cert\") pod \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\" (UID: \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\") "
Sep 17 20:52:34.443875 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:34.443771    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-console-oauth-config\") pod \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\" (UID: \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\") "
Sep 17 20:52:34.443875 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:34.443797    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-console-config\") pod \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\" (UID: \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\") "
Sep 17 20:52:34.443875 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:34.443832    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-trusted-ca-bundle\") pod \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\" (UID: \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\") "
Sep 17 20:52:34.443875 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:34.443866    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-console-serving-cert\") pod \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\" (UID: \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\") "
Sep 17 20:52:34.444074 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:34.443891    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tkq7\" (UniqueName: \"kubernetes.io/projected/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-kube-api-access-9tkq7\") pod \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\" (UID: \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\") "
Sep 17 20:52:34.444074 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:34.443918    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-service-ca\") pod \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\" (UID: \"57891b1c-80f0-489b-a6d6-c0ca2287ccb2\") "
Sep 17 20:52:34.444177 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:34.444085    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "57891b1c-80f0-489b-a6d6-c0ca2287ccb2" (UID: "57891b1c-80f0-489b-a6d6-c0ca2287ccb2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:52:34.444320 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:34.444297    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-oauth-serving-cert\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:52:34.444422 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:34.444354    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-console-config" (OuterVolumeSpecName: "console-config") pod "57891b1c-80f0-489b-a6d6-c0ca2287ccb2" (UID: "57891b1c-80f0-489b-a6d6-c0ca2287ccb2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:52:34.444422 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:34.444369    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-service-ca" (OuterVolumeSpecName: "service-ca") pod "57891b1c-80f0-489b-a6d6-c0ca2287ccb2" (UID: "57891b1c-80f0-489b-a6d6-c0ca2287ccb2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:52:34.444514 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:34.444427    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "57891b1c-80f0-489b-a6d6-c0ca2287ccb2" (UID: "57891b1c-80f0-489b-a6d6-c0ca2287ccb2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:52:34.457491 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:34.457470    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "57891b1c-80f0-489b-a6d6-c0ca2287ccb2" (UID: "57891b1c-80f0-489b-a6d6-c0ca2287ccb2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:52:34.466507 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:34.466482    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "57891b1c-80f0-489b-a6d6-c0ca2287ccb2" (UID: "57891b1c-80f0-489b-a6d6-c0ca2287ccb2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:52:34.466596 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:34.466532    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-kube-api-access-9tkq7" (OuterVolumeSpecName: "kube-api-access-9tkq7") pod "57891b1c-80f0-489b-a6d6-c0ca2287ccb2" (UID: "57891b1c-80f0-489b-a6d6-c0ca2287ccb2"). InnerVolumeSpecName "kube-api-access-9tkq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:52:34.544944 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:34.544924    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-console-oauth-config\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:52:34.544944 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:34.544944    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-console-config\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:52:34.545084 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:34.544953    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-trusted-ca-bundle\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:52:34.545084 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:34.544961    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-console-serving-cert\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:52:34.545084 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:34.544970    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tkq7\" (UniqueName: \"kubernetes.io/projected/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-kube-api-access-9tkq7\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:52:34.545084 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:34.544978    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57891b1c-80f0-489b-a6d6-c0ca2287ccb2-service-ca\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:52:35.164123 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:35.164088    2566 generic.go:356] "Generic (PLEG): container finished" podID="aa1dd337-94f0-4707-a264-577fa255c317" containerID="96dcc6984eb8ab9294f22fc96dcfbd72e4ee09c513ab5b5ca8a4159b0ebc10d8" exitCode=0
Sep 17 20:52:35.164571 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:35.164160    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8dg4p" event={"ID":"aa1dd337-94f0-4707-a264-577fa255c317","Type":"ContainerDied","Data":"96dcc6984eb8ab9294f22fc96dcfbd72e4ee09c513ab5b5ca8a4159b0ebc10d8"}
Sep 17 20:52:35.164571 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:35.164514    2566 scope.go:117] "RemoveContainer" containerID="96dcc6984eb8ab9294f22fc96dcfbd72e4ee09c513ab5b5ca8a4159b0ebc10d8"
Sep 17 20:52:35.165522 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:35.165503    2566 generic.go:356] "Generic (PLEG): container finished" podID="57891b1c-80f0-489b-a6d6-c0ca2287ccb2" containerID="949a734ddf74f21af4aabd097bde724d36522c7b1660337764ecb66402373789" exitCode=0
Sep 17 20:52:35.165610 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:35.165532    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-755b7db98d-9t8hg" event={"ID":"57891b1c-80f0-489b-a6d6-c0ca2287ccb2","Type":"ContainerDied","Data":"949a734ddf74f21af4aabd097bde724d36522c7b1660337764ecb66402373789"}
Sep 17 20:52:35.165610 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:35.165550    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-755b7db98d-9t8hg"
Sep 17 20:52:35.165610 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:35.165572    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-755b7db98d-9t8hg" event={"ID":"57891b1c-80f0-489b-a6d6-c0ca2287ccb2","Type":"ContainerDied","Data":"a1ef5edfd4ddb04f403cb63cf410cdd13dba4d0b19730283eb360e7c623322ca"}
Sep 17 20:52:35.165610 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:35.165591    2566 scope.go:117] "RemoveContainer" containerID="949a734ddf74f21af4aabd097bde724d36522c7b1660337764ecb66402373789"
Sep 17 20:52:35.175605 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:35.175590    2566 scope.go:117] "RemoveContainer" containerID="949a734ddf74f21af4aabd097bde724d36522c7b1660337764ecb66402373789"
Sep 17 20:52:35.175887 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:52:35.175864    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"949a734ddf74f21af4aabd097bde724d36522c7b1660337764ecb66402373789\": container with ID starting with 949a734ddf74f21af4aabd097bde724d36522c7b1660337764ecb66402373789 not found: ID does not exist" containerID="949a734ddf74f21af4aabd097bde724d36522c7b1660337764ecb66402373789"
Sep 17 20:52:35.175934 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:35.175898    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"949a734ddf74f21af4aabd097bde724d36522c7b1660337764ecb66402373789"} err="failed to get container status \"949a734ddf74f21af4aabd097bde724d36522c7b1660337764ecb66402373789\": rpc error: code = NotFound desc = could not find container \"949a734ddf74f21af4aabd097bde724d36522c7b1660337764ecb66402373789\": container with ID starting with 949a734ddf74f21af4aabd097bde724d36522c7b1660337764ecb66402373789 not found: ID does not exist"
Sep 17 20:52:35.210750 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:35.210726    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-755b7db98d-9t8hg"]
Sep 17 20:52:35.216736 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:35.216719    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-755b7db98d-9t8hg"]
Sep 17 20:52:36.170371 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:36.170334    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8dg4p" event={"ID":"aa1dd337-94f0-4707-a264-577fa255c317","Type":"ContainerStarted","Data":"14ea6192b8c78e98e57f3358f49e46ed1a6250f99ff2a67602ff58d66bd378a0"}
Sep 17 20:52:36.458578 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:52:36.458505    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57891b1c-80f0-489b-a6d6-c0ca2287ccb2" path="/var/lib/kubelet/pods/57891b1c-80f0-489b-a6d6-c0ca2287ccb2/volumes"
Sep 17 20:53:21.432219 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:21.432177    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 20:53:21.432613 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:21.432432    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerName="alertmanager" containerID="cri-o://5f1fbb1b6581521647a2c27e05afcaa1a34b9458dd7de9f6e826b78fd8ce1f47" gracePeriod=120
Sep 17 20:53:21.432613 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:21.432513    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerName="kube-rbac-proxy-web" containerID="cri-o://ed264130222cf71c748308eab5a5bfea546aca339e0b3df8fb5806ce8af79beb" gracePeriod=120
Sep 17 20:53:21.432613 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:21.432506    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerName="kube-rbac-proxy-metric" containerID="cri-o://7db8b1255759bd301781b7f87222a4ebef786447d6320d6c22c98b3d3501f3f0" gracePeriod=120
Sep 17 20:53:21.432613 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:21.432542    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerName="kube-rbac-proxy" containerID="cri-o://c5eae3cd250ec50d6d607ddf7eda3b7d78271cc2f0c62c2c6e956a45a5b80ad8" gracePeriod=120
Sep 17 20:53:21.432613 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:21.432581    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerName="prom-label-proxy" containerID="cri-o://731b49cb3d2244665f6c31aee09ba637cd096d32f8924c7189afd0de00118ce7" gracePeriod=120
Sep 17 20:53:21.432837 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:21.432616    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerName="config-reloader" containerID="cri-o://7d204c09ef813ff3b720df3d5757984346d9655189a033dfb47f417f65a2fb79" gracePeriod=120
Sep 17 20:53:22.343010 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.342980    2566 generic.go:356] "Generic (PLEG): container finished" podID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerID="731b49cb3d2244665f6c31aee09ba637cd096d32f8924c7189afd0de00118ce7" exitCode=0
Sep 17 20:53:22.343010 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.343004    2566 generic.go:356] "Generic (PLEG): container finished" podID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerID="c5eae3cd250ec50d6d607ddf7eda3b7d78271cc2f0c62c2c6e956a45a5b80ad8" exitCode=0
Sep 17 20:53:22.343010 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.343011    2566 generic.go:356] "Generic (PLEG): container finished" podID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerID="7d204c09ef813ff3b720df3d5757984346d9655189a033dfb47f417f65a2fb79" exitCode=0
Sep 17 20:53:22.343010 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.343016    2566 generic.go:356] "Generic (PLEG): container finished" podID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerID="5f1fbb1b6581521647a2c27e05afcaa1a34b9458dd7de9f6e826b78fd8ce1f47" exitCode=0
Sep 17 20:53:22.343277 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.343047    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91ea8785-e65b-4c0c-bbb7-e5514d627965","Type":"ContainerDied","Data":"731b49cb3d2244665f6c31aee09ba637cd096d32f8924c7189afd0de00118ce7"}
Sep 17 20:53:22.343277 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.343108    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91ea8785-e65b-4c0c-bbb7-e5514d627965","Type":"ContainerDied","Data":"c5eae3cd250ec50d6d607ddf7eda3b7d78271cc2f0c62c2c6e956a45a5b80ad8"}
Sep 17 20:53:22.343277 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.343134    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91ea8785-e65b-4c0c-bbb7-e5514d627965","Type":"ContainerDied","Data":"7d204c09ef813ff3b720df3d5757984346d9655189a033dfb47f417f65a2fb79"}
Sep 17 20:53:22.343277 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.343157    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91ea8785-e65b-4c0c-bbb7-e5514d627965","Type":"ContainerDied","Data":"5f1fbb1b6581521647a2c27e05afcaa1a34b9458dd7de9f6e826b78fd8ce1f47"}
Sep 17 20:53:22.673382 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.673355    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:22.804844 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.804801    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/91ea8785-e65b-4c0c-bbb7-e5514d627965-tls-assets\") pod \"91ea8785-e65b-4c0c-bbb7-e5514d627965\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") "
Sep 17 20:53:22.804844 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.804853    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lphvc\" (UniqueName: \"kubernetes.io/projected/91ea8785-e65b-4c0c-bbb7-e5514d627965-kube-api-access-lphvc\") pod \"91ea8785-e65b-4c0c-bbb7-e5514d627965\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") "
Sep 17 20:53:22.805084 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.804879    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-cluster-tls-config\") pod \"91ea8785-e65b-4c0c-bbb7-e5514d627965\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") "
Sep 17 20:53:22.805084 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.804916    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-secret-alertmanager-kube-rbac-proxy-web\") pod \"91ea8785-e65b-4c0c-bbb7-e5514d627965\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") "
Sep 17 20:53:22.805084 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.804943    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-secret-alertmanager-kube-rbac-proxy-metric\") pod \"91ea8785-e65b-4c0c-bbb7-e5514d627965\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") "
Sep 17 20:53:22.805084 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.804990    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-secret-alertmanager-kube-rbac-proxy\") pod \"91ea8785-e65b-4c0c-bbb7-e5514d627965\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") "
Sep 17 20:53:22.805084 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.805016    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/91ea8785-e65b-4c0c-bbb7-e5514d627965-config-out\") pod \"91ea8785-e65b-4c0c-bbb7-e5514d627965\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") "
Sep 17 20:53:22.805084 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.805049    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91ea8785-e65b-4c0c-bbb7-e5514d627965-metrics-client-ca\") pod \"91ea8785-e65b-4c0c-bbb7-e5514d627965\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") "
Sep 17 20:53:22.805532 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.805469    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ea8785-e65b-4c0c-bbb7-e5514d627965-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "91ea8785-e65b-4c0c-bbb7-e5514d627965" (UID: "91ea8785-e65b-4c0c-bbb7-e5514d627965"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:53:22.805977 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.805683    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-secret-alertmanager-main-tls\") pod \"91ea8785-e65b-4c0c-bbb7-e5514d627965\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") "
Sep 17 20:53:22.805977 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.805729    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/91ea8785-e65b-4c0c-bbb7-e5514d627965-alertmanager-main-db\") pod \"91ea8785-e65b-4c0c-bbb7-e5514d627965\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") "
Sep 17 20:53:22.805977 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.805768    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91ea8785-e65b-4c0c-bbb7-e5514d627965-alertmanager-trusted-ca-bundle\") pod \"91ea8785-e65b-4c0c-bbb7-e5514d627965\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") "
Sep 17 20:53:22.805977 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.805794    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-web-config\") pod \"91ea8785-e65b-4c0c-bbb7-e5514d627965\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") "
Sep 17 20:53:22.805977 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.805826    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-config-volume\") pod \"91ea8785-e65b-4c0c-bbb7-e5514d627965\" (UID: \"91ea8785-e65b-4c0c-bbb7-e5514d627965\") "
Sep 17 20:53:22.806409 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.806271    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ea8785-e65b-4c0c-bbb7-e5514d627965-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "91ea8785-e65b-4c0c-bbb7-e5514d627965" (UID: "91ea8785-e65b-4c0c-bbb7-e5514d627965"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:53:22.806409 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.806286    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ea8785-e65b-4c0c-bbb7-e5514d627965-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "91ea8785-e65b-4c0c-bbb7-e5514d627965" (UID: "91ea8785-e65b-4c0c-bbb7-e5514d627965"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:53:22.806760 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.806734    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91ea8785-e65b-4c0c-bbb7-e5514d627965-metrics-client-ca\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:53:22.806852 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.806766    2566 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/91ea8785-e65b-4c0c-bbb7-e5514d627965-alertmanager-main-db\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:53:22.806852 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.806784    2566 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91ea8785-e65b-4c0c-bbb7-e5514d627965-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:53:22.808005 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.807973    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "91ea8785-e65b-4c0c-bbb7-e5514d627965" (UID: "91ea8785-e65b-4c0c-bbb7-e5514d627965"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:53:22.808113 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.808063    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ea8785-e65b-4c0c-bbb7-e5514d627965-kube-api-access-lphvc" (OuterVolumeSpecName: "kube-api-access-lphvc") pod "91ea8785-e65b-4c0c-bbb7-e5514d627965" (UID: "91ea8785-e65b-4c0c-bbb7-e5514d627965"). InnerVolumeSpecName "kube-api-access-lphvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:53:22.808113 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.808065    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "91ea8785-e65b-4c0c-bbb7-e5514d627965" (UID: "91ea8785-e65b-4c0c-bbb7-e5514d627965"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:53:22.808441 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.808417    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ea8785-e65b-4c0c-bbb7-e5514d627965-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "91ea8785-e65b-4c0c-bbb7-e5514d627965" (UID: "91ea8785-e65b-4c0c-bbb7-e5514d627965"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:53:22.808794 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.808768    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ea8785-e65b-4c0c-bbb7-e5514d627965-config-out" (OuterVolumeSpecName: "config-out") pod "91ea8785-e65b-4c0c-bbb7-e5514d627965" (UID: "91ea8785-e65b-4c0c-bbb7-e5514d627965"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:53:22.808880 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.808830    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "91ea8785-e65b-4c0c-bbb7-e5514d627965" (UID: "91ea8785-e65b-4c0c-bbb7-e5514d627965"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:53:22.808930 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.808867    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "91ea8785-e65b-4c0c-bbb7-e5514d627965" (UID: "91ea8785-e65b-4c0c-bbb7-e5514d627965"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:53:22.809540 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.809522    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-config-volume" (OuterVolumeSpecName: "config-volume") pod "91ea8785-e65b-4c0c-bbb7-e5514d627965" (UID: "91ea8785-e65b-4c0c-bbb7-e5514d627965"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:53:22.812918 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.812888    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "91ea8785-e65b-4c0c-bbb7-e5514d627965" (UID: "91ea8785-e65b-4c0c-bbb7-e5514d627965"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:53:22.818787 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.818768    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-web-config" (OuterVolumeSpecName: "web-config") pod "91ea8785-e65b-4c0c-bbb7-e5514d627965" (UID: "91ea8785-e65b-4c0c-bbb7-e5514d627965"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:53:22.907094 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.907027    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-secret-alertmanager-main-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:53:22.907094 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.907051    2566 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-web-config\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:53:22.907094 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.907061    2566 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-config-volume\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:53:22.907094 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.907069    2566 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/91ea8785-e65b-4c0c-bbb7-e5514d627965-tls-assets\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:53:22.907094 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.907076    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lphvc\" (UniqueName: \"kubernetes.io/projected/91ea8785-e65b-4c0c-bbb7-e5514d627965-kube-api-access-lphvc\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:53:22.907094 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.907085    2566 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-cluster-tls-config\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:53:22.907094 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.907095    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:53:22.907396 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.907105    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:53:22.907396 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.907114    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91ea8785-e65b-4c0c-bbb7-e5514d627965-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:53:22.907396 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:22.907122    2566 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/91ea8785-e65b-4c0c-bbb7-e5514d627965-config-out\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:53:23.349240 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.349195    2566 generic.go:356] "Generic (PLEG): container finished" podID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerID="7db8b1255759bd301781b7f87222a4ebef786447d6320d6c22c98b3d3501f3f0" exitCode=0
Sep 17 20:53:23.349240 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.349233    2566 generic.go:356] "Generic (PLEG): container finished" podID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerID="ed264130222cf71c748308eab5a5bfea546aca339e0b3df8fb5806ce8af79beb" exitCode=0
Sep 17 20:53:23.349484 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.349245    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91ea8785-e65b-4c0c-bbb7-e5514d627965","Type":"ContainerDied","Data":"7db8b1255759bd301781b7f87222a4ebef786447d6320d6c22c98b3d3501f3f0"}
Sep 17 20:53:23.349484 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.349318    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.349484 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.349338    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91ea8785-e65b-4c0c-bbb7-e5514d627965","Type":"ContainerDied","Data":"ed264130222cf71c748308eab5a5bfea546aca339e0b3df8fb5806ce8af79beb"}
Sep 17 20:53:23.349484 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.349380    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91ea8785-e65b-4c0c-bbb7-e5514d627965","Type":"ContainerDied","Data":"be5da4ac3ad701fe8b2c9ec4a1d040755da52981e58ac54eb23eaf89855df38a"}
Sep 17 20:53:23.349484 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.349403    2566 scope.go:117] "RemoveContainer" containerID="731b49cb3d2244665f6c31aee09ba637cd096d32f8924c7189afd0de00118ce7"
Sep 17 20:53:23.357158 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.357136    2566 scope.go:117] "RemoveContainer" containerID="7db8b1255759bd301781b7f87222a4ebef786447d6320d6c22c98b3d3501f3f0"
Sep 17 20:53:23.363656 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.363642    2566 scope.go:117] "RemoveContainer" containerID="c5eae3cd250ec50d6d607ddf7eda3b7d78271cc2f0c62c2c6e956a45a5b80ad8"
Sep 17 20:53:23.370098 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.370078    2566 scope.go:117] "RemoveContainer" containerID="ed264130222cf71c748308eab5a5bfea546aca339e0b3df8fb5806ce8af79beb"
Sep 17 20:53:23.375906 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.375887    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 20:53:23.376561 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.376547    2566 scope.go:117] "RemoveContainer" containerID="7d204c09ef813ff3b720df3d5757984346d9655189a033dfb47f417f65a2fb79"
Sep 17 20:53:23.380002 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.379979    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 20:53:23.383441 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.383426    2566 scope.go:117] "RemoveContainer" containerID="5f1fbb1b6581521647a2c27e05afcaa1a34b9458dd7de9f6e826b78fd8ce1f47"
Sep 17 20:53:23.389836 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.389820    2566 scope.go:117] "RemoveContainer" containerID="7c2373ad7c629feb54a23ba725230135fd1123b803aa16c858714713b1d0d5cd"
Sep 17 20:53:23.396002 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.395981    2566 scope.go:117] "RemoveContainer" containerID="731b49cb3d2244665f6c31aee09ba637cd096d32f8924c7189afd0de00118ce7"
Sep 17 20:53:23.396391 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:53:23.396267    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"731b49cb3d2244665f6c31aee09ba637cd096d32f8924c7189afd0de00118ce7\": container with ID starting with 731b49cb3d2244665f6c31aee09ba637cd096d32f8924c7189afd0de00118ce7 not found: ID does not exist" containerID="731b49cb3d2244665f6c31aee09ba637cd096d32f8924c7189afd0de00118ce7"
Sep 17 20:53:23.396391 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.396310    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"731b49cb3d2244665f6c31aee09ba637cd096d32f8924c7189afd0de00118ce7"} err="failed to get container status \"731b49cb3d2244665f6c31aee09ba637cd096d32f8924c7189afd0de00118ce7\": rpc error: code = NotFound desc = could not find container \"731b49cb3d2244665f6c31aee09ba637cd096d32f8924c7189afd0de00118ce7\": container with ID starting with 731b49cb3d2244665f6c31aee09ba637cd096d32f8924c7189afd0de00118ce7 not found: ID does not exist"
Sep 17 20:53:23.396391 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.396334    2566 scope.go:117] "RemoveContainer" containerID="7db8b1255759bd301781b7f87222a4ebef786447d6320d6c22c98b3d3501f3f0"
Sep 17 20:53:23.396611 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:53:23.396593    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7db8b1255759bd301781b7f87222a4ebef786447d6320d6c22c98b3d3501f3f0\": container with ID starting with 7db8b1255759bd301781b7f87222a4ebef786447d6320d6c22c98b3d3501f3f0 not found: ID does not exist" containerID="7db8b1255759bd301781b7f87222a4ebef786447d6320d6c22c98b3d3501f3f0"
Sep 17 20:53:23.396649 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.396615    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7db8b1255759bd301781b7f87222a4ebef786447d6320d6c22c98b3d3501f3f0"} err="failed to get container status \"7db8b1255759bd301781b7f87222a4ebef786447d6320d6c22c98b3d3501f3f0\": rpc error: code = NotFound desc = could not find container \"7db8b1255759bd301781b7f87222a4ebef786447d6320d6c22c98b3d3501f3f0\": container with ID starting with 7db8b1255759bd301781b7f87222a4ebef786447d6320d6c22c98b3d3501f3f0 not found: ID does not exist"
Sep 17 20:53:23.396649 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.396631    2566 scope.go:117] "RemoveContainer" containerID="c5eae3cd250ec50d6d607ddf7eda3b7d78271cc2f0c62c2c6e956a45a5b80ad8"
Sep 17 20:53:23.396869 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:53:23.396852    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5eae3cd250ec50d6d607ddf7eda3b7d78271cc2f0c62c2c6e956a45a5b80ad8\": container with ID starting with c5eae3cd250ec50d6d607ddf7eda3b7d78271cc2f0c62c2c6e956a45a5b80ad8 not found: ID does not exist" containerID="c5eae3cd250ec50d6d607ddf7eda3b7d78271cc2f0c62c2c6e956a45a5b80ad8"
Sep 17 20:53:23.396933 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.396876    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5eae3cd250ec50d6d607ddf7eda3b7d78271cc2f0c62c2c6e956a45a5b80ad8"} err="failed to get container status \"c5eae3cd250ec50d6d607ddf7eda3b7d78271cc2f0c62c2c6e956a45a5b80ad8\": rpc error: code = NotFound desc = could not find container \"c5eae3cd250ec50d6d607ddf7eda3b7d78271cc2f0c62c2c6e956a45a5b80ad8\": container with ID starting with c5eae3cd250ec50d6d607ddf7eda3b7d78271cc2f0c62c2c6e956a45a5b80ad8 not found: ID does not exist"
Sep 17 20:53:23.396933 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.396897    2566 scope.go:117] "RemoveContainer" containerID="ed264130222cf71c748308eab5a5bfea546aca339e0b3df8fb5806ce8af79beb"
Sep 17 20:53:23.397114 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:53:23.397100    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed264130222cf71c748308eab5a5bfea546aca339e0b3df8fb5806ce8af79beb\": container with ID starting with ed264130222cf71c748308eab5a5bfea546aca339e0b3df8fb5806ce8af79beb not found: ID does not exist" containerID="ed264130222cf71c748308eab5a5bfea546aca339e0b3df8fb5806ce8af79beb"
Sep 17 20:53:23.397151 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.397116    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed264130222cf71c748308eab5a5bfea546aca339e0b3df8fb5806ce8af79beb"} err="failed to get container status \"ed264130222cf71c748308eab5a5bfea546aca339e0b3df8fb5806ce8af79beb\": rpc error: code = NotFound desc = could not find container \"ed264130222cf71c748308eab5a5bfea546aca339e0b3df8fb5806ce8af79beb\": container with ID starting with ed264130222cf71c748308eab5a5bfea546aca339e0b3df8fb5806ce8af79beb not found: ID does not exist"
Sep 17 20:53:23.397151 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.397128    2566 scope.go:117] "RemoveContainer" containerID="7d204c09ef813ff3b720df3d5757984346d9655189a033dfb47f417f65a2fb79"
Sep 17 20:53:23.397362 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:53:23.397346    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d204c09ef813ff3b720df3d5757984346d9655189a033dfb47f417f65a2fb79\": container with ID starting with 7d204c09ef813ff3b720df3d5757984346d9655189a033dfb47f417f65a2fb79 not found: ID does not exist" containerID="7d204c09ef813ff3b720df3d5757984346d9655189a033dfb47f417f65a2fb79"
Sep 17 20:53:23.397409 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.397366    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d204c09ef813ff3b720df3d5757984346d9655189a033dfb47f417f65a2fb79"} err="failed to get container status \"7d204c09ef813ff3b720df3d5757984346d9655189a033dfb47f417f65a2fb79\": rpc error: code = NotFound desc = could not find container \"7d204c09ef813ff3b720df3d5757984346d9655189a033dfb47f417f65a2fb79\": container with ID starting with 7d204c09ef813ff3b720df3d5757984346d9655189a033dfb47f417f65a2fb79 not found: ID does not exist"
Sep 17 20:53:23.397409 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.397380    2566 scope.go:117] "RemoveContainer" containerID="5f1fbb1b6581521647a2c27e05afcaa1a34b9458dd7de9f6e826b78fd8ce1f47"
Sep 17 20:53:23.397578 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:53:23.397562    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f1fbb1b6581521647a2c27e05afcaa1a34b9458dd7de9f6e826b78fd8ce1f47\": container with ID starting with 5f1fbb1b6581521647a2c27e05afcaa1a34b9458dd7de9f6e826b78fd8ce1f47 not found: ID does not exist" containerID="5f1fbb1b6581521647a2c27e05afcaa1a34b9458dd7de9f6e826b78fd8ce1f47"
Sep 17 20:53:23.397637 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.397585    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f1fbb1b6581521647a2c27e05afcaa1a34b9458dd7de9f6e826b78fd8ce1f47"} err="failed to get container status \"5f1fbb1b6581521647a2c27e05afcaa1a34b9458dd7de9f6e826b78fd8ce1f47\": rpc error: code = NotFound desc = could not find container \"5f1fbb1b6581521647a2c27e05afcaa1a34b9458dd7de9f6e826b78fd8ce1f47\": container with ID starting with 5f1fbb1b6581521647a2c27e05afcaa1a34b9458dd7de9f6e826b78fd8ce1f47 not found: ID does not exist"
Sep 17 20:53:23.397637 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.397603    2566 scope.go:117] "RemoveContainer" containerID="7c2373ad7c629feb54a23ba725230135fd1123b803aa16c858714713b1d0d5cd"
Sep 17 20:53:23.397815 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:53:23.397794    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c2373ad7c629feb54a23ba725230135fd1123b803aa16c858714713b1d0d5cd\": container with ID starting with 7c2373ad7c629feb54a23ba725230135fd1123b803aa16c858714713b1d0d5cd not found: ID does not exist" containerID="7c2373ad7c629feb54a23ba725230135fd1123b803aa16c858714713b1d0d5cd"
Sep 17 20:53:23.397848 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.397818    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c2373ad7c629feb54a23ba725230135fd1123b803aa16c858714713b1d0d5cd"} err="failed to get container status \"7c2373ad7c629feb54a23ba725230135fd1123b803aa16c858714713b1d0d5cd\": rpc error: code = NotFound desc = could not find container \"7c2373ad7c629feb54a23ba725230135fd1123b803aa16c858714713b1d0d5cd\": container with ID starting with 7c2373ad7c629feb54a23ba725230135fd1123b803aa16c858714713b1d0d5cd not found: ID does not exist"
Sep 17 20:53:23.397848 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.397830    2566 scope.go:117] "RemoveContainer" containerID="731b49cb3d2244665f6c31aee09ba637cd096d32f8924c7189afd0de00118ce7"
Sep 17 20:53:23.398040 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.398024    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"731b49cb3d2244665f6c31aee09ba637cd096d32f8924c7189afd0de00118ce7"} err="failed to get container status \"731b49cb3d2244665f6c31aee09ba637cd096d32f8924c7189afd0de00118ce7\": rpc error: code = NotFound desc = could not find container \"731b49cb3d2244665f6c31aee09ba637cd096d32f8924c7189afd0de00118ce7\": container with ID starting with 731b49cb3d2244665f6c31aee09ba637cd096d32f8924c7189afd0de00118ce7 not found: ID does not exist"
Sep 17 20:53:23.398105 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.398041    2566 scope.go:117] "RemoveContainer" containerID="7db8b1255759bd301781b7f87222a4ebef786447d6320d6c22c98b3d3501f3f0"
Sep 17 20:53:23.398261 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.398241    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7db8b1255759bd301781b7f87222a4ebef786447d6320d6c22c98b3d3501f3f0"} err="failed to get container status \"7db8b1255759bd301781b7f87222a4ebef786447d6320d6c22c98b3d3501f3f0\": rpc error: code = NotFound desc = could not find container \"7db8b1255759bd301781b7f87222a4ebef786447d6320d6c22c98b3d3501f3f0\": container with ID starting with 7db8b1255759bd301781b7f87222a4ebef786447d6320d6c22c98b3d3501f3f0 not found: ID does not exist"
Sep 17 20:53:23.398261 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.398258    2566 scope.go:117] "RemoveContainer" containerID="c5eae3cd250ec50d6d607ddf7eda3b7d78271cc2f0c62c2c6e956a45a5b80ad8"
Sep 17 20:53:23.398518 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.398498    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5eae3cd250ec50d6d607ddf7eda3b7d78271cc2f0c62c2c6e956a45a5b80ad8"} err="failed to get container status \"c5eae3cd250ec50d6d607ddf7eda3b7d78271cc2f0c62c2c6e956a45a5b80ad8\": rpc error: code = NotFound desc = could not find container \"c5eae3cd250ec50d6d607ddf7eda3b7d78271cc2f0c62c2c6e956a45a5b80ad8\": container with ID starting with c5eae3cd250ec50d6d607ddf7eda3b7d78271cc2f0c62c2c6e956a45a5b80ad8 not found: ID does not exist"
Sep 17 20:53:23.398518 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.398517    2566 scope.go:117] "RemoveContainer" containerID="ed264130222cf71c748308eab5a5bfea546aca339e0b3df8fb5806ce8af79beb"
Sep 17 20:53:23.398750 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.398733    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed264130222cf71c748308eab5a5bfea546aca339e0b3df8fb5806ce8af79beb"} err="failed to get container status \"ed264130222cf71c748308eab5a5bfea546aca339e0b3df8fb5806ce8af79beb\": rpc error: code = NotFound desc = could not find container \"ed264130222cf71c748308eab5a5bfea546aca339e0b3df8fb5806ce8af79beb\": container with ID starting with ed264130222cf71c748308eab5a5bfea546aca339e0b3df8fb5806ce8af79beb not found: ID does not exist"
Sep 17 20:53:23.398796 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.398751    2566 scope.go:117] "RemoveContainer" containerID="7d204c09ef813ff3b720df3d5757984346d9655189a033dfb47f417f65a2fb79"
Sep 17 20:53:23.398962 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.398946    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d204c09ef813ff3b720df3d5757984346d9655189a033dfb47f417f65a2fb79"} err="failed to get container status \"7d204c09ef813ff3b720df3d5757984346d9655189a033dfb47f417f65a2fb79\": rpc error: code = NotFound desc = could not find container \"7d204c09ef813ff3b720df3d5757984346d9655189a033dfb47f417f65a2fb79\": container with ID starting with 7d204c09ef813ff3b720df3d5757984346d9655189a033dfb47f417f65a2fb79 not found: ID does not exist"
Sep 17 20:53:23.398962 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.398961    2566 scope.go:117] "RemoveContainer" containerID="5f1fbb1b6581521647a2c27e05afcaa1a34b9458dd7de9f6e826b78fd8ce1f47"
Sep 17 20:53:23.399146 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.399131    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f1fbb1b6581521647a2c27e05afcaa1a34b9458dd7de9f6e826b78fd8ce1f47"} err="failed to get container status \"5f1fbb1b6581521647a2c27e05afcaa1a34b9458dd7de9f6e826b78fd8ce1f47\": rpc error: code = NotFound desc = could not find container \"5f1fbb1b6581521647a2c27e05afcaa1a34b9458dd7de9f6e826b78fd8ce1f47\": container with ID starting with 5f1fbb1b6581521647a2c27e05afcaa1a34b9458dd7de9f6e826b78fd8ce1f47 not found: ID does not exist"
Sep 17 20:53:23.399146 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.399146    2566 scope.go:117] "RemoveContainer" containerID="7c2373ad7c629feb54a23ba725230135fd1123b803aa16c858714713b1d0d5cd"
Sep 17 20:53:23.399368 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.399353    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c2373ad7c629feb54a23ba725230135fd1123b803aa16c858714713b1d0d5cd"} err="failed to get container status \"7c2373ad7c629feb54a23ba725230135fd1123b803aa16c858714713b1d0d5cd\": rpc error: code = NotFound desc = could not find container \"7c2373ad7c629feb54a23ba725230135fd1123b803aa16c858714713b1d0d5cd\": container with ID starting with 7c2373ad7c629feb54a23ba725230135fd1123b803aa16c858714713b1d0d5cd not found: ID does not exist"
Sep 17 20:53:23.405557 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.405539    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 20:53:23.405768 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.405757    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerName="init-config-reloader"
Sep 17 20:53:23.405824 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.405770    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerName="init-config-reloader"
Sep 17 20:53:23.405824 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.405799    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerName="kube-rbac-proxy-web"
Sep 17 20:53:23.405824 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.405805    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerName="kube-rbac-proxy-web"
Sep 17 20:53:23.405824 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.405814    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerName="kube-rbac-proxy-metric"
Sep 17 20:53:23.405824 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.405819    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerName="kube-rbac-proxy-metric"
Sep 17 20:53:23.405964 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.405826    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57891b1c-80f0-489b-a6d6-c0ca2287ccb2" containerName="console"
Sep 17 20:53:23.405964 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.405832    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="57891b1c-80f0-489b-a6d6-c0ca2287ccb2" containerName="console"
Sep 17 20:53:23.405964 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.405838    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerName="config-reloader"
Sep 17 20:53:23.405964 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.405843    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerName="config-reloader"
Sep 17 20:53:23.405964 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.405851    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerName="alertmanager"
Sep 17 20:53:23.405964 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.405857    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerName="alertmanager"
Sep 17 20:53:23.405964 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.405862    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerName="kube-rbac-proxy"
Sep 17 20:53:23.405964 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.405867    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerName="kube-rbac-proxy"
Sep 17 20:53:23.405964 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.405874    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1552a7fa-597f-406a-8c77-cca66297356d" containerName="registry"
Sep 17 20:53:23.405964 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.405879    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1552a7fa-597f-406a-8c77-cca66297356d" containerName="registry"
Sep 17 20:53:23.405964 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.405886    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerName="prom-label-proxy"
Sep 17 20:53:23.405964 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.405892    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerName="prom-label-proxy"
Sep 17 20:53:23.405964 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.405930    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerName="kube-rbac-proxy-web"
Sep 17 20:53:23.405964 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.405937    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1552a7fa-597f-406a-8c77-cca66297356d" containerName="registry"
Sep 17 20:53:23.405964 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.405942    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="57891b1c-80f0-489b-a6d6-c0ca2287ccb2" containerName="console"
Sep 17 20:53:23.405964 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.405948    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerName="alertmanager"
Sep 17 20:53:23.405964 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.405953    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerName="kube-rbac-proxy-metric"
Sep 17 20:53:23.405964 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.405959    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerName="prom-label-proxy"
Sep 17 20:53:23.405964 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.405965    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerName="config-reloader"
Sep 17 20:53:23.405964 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.405970    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="91ea8785-e65b-4c0c-bbb7-e5514d627965" containerName="kube-rbac-proxy"
Sep 17 20:53:23.410820 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.410803    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.413446 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.413430    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 20:53:23.413542 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.413433    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 20:53:23.413542 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.413481    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-58dfl\""
Sep 17 20:53:23.413542 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.413511    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 20:53:23.413924 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.413906    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 20:53:23.413924 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.413914    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 20:53:23.413924 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.413924    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 20:53:23.414105 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.413949    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 20:53:23.414105 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.414025    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 20:53:23.420558 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.420541    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 20:53:23.427000 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.426980    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 20:53:23.511989 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.511967    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bf415666-9904-4e67-a1cd-2ad770aa292f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.512099 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.512017    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfv8z\" (UniqueName: \"kubernetes.io/projected/bf415666-9904-4e67-a1cd-2ad770aa292f-kube-api-access-rfv8z\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.512099 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.512048    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bf415666-9904-4e67-a1cd-2ad770aa292f-web-config\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.512271 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.512099    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf415666-9904-4e67-a1cd-2ad770aa292f-config-out\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.512271 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.512149    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf415666-9904-4e67-a1cd-2ad770aa292f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.512271 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.512186    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf415666-9904-4e67-a1cd-2ad770aa292f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.512271 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.512222    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bf415666-9904-4e67-a1cd-2ad770aa292f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.512271 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.512238    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf415666-9904-4e67-a1cd-2ad770aa292f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.512271 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.512256    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bf415666-9904-4e67-a1cd-2ad770aa292f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.512488 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.512305    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bf415666-9904-4e67-a1cd-2ad770aa292f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.512488 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.512350    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bf415666-9904-4e67-a1cd-2ad770aa292f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.512488 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.512369    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bf415666-9904-4e67-a1cd-2ad770aa292f-config-volume\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.512488 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.512395    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf415666-9904-4e67-a1cd-2ad770aa292f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.612963 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.612912    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf415666-9904-4e67-a1cd-2ad770aa292f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.612963 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.612943    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bf415666-9904-4e67-a1cd-2ad770aa292f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.613125 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.612971    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfv8z\" (UniqueName: \"kubernetes.io/projected/bf415666-9904-4e67-a1cd-2ad770aa292f-kube-api-access-rfv8z\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.613125 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.612996    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bf415666-9904-4e67-a1cd-2ad770aa292f-web-config\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.613125 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.613019    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf415666-9904-4e67-a1cd-2ad770aa292f-config-out\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.613125 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.613053    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf415666-9904-4e67-a1cd-2ad770aa292f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.613125 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.613081    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf415666-9904-4e67-a1cd-2ad770aa292f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.613125 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.613107    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bf415666-9904-4e67-a1cd-2ad770aa292f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.613412 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.613129    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf415666-9904-4e67-a1cd-2ad770aa292f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.613412 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.613154    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bf415666-9904-4e67-a1cd-2ad770aa292f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.613412 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.613177    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bf415666-9904-4e67-a1cd-2ad770aa292f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.613412 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.613232    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bf415666-9904-4e67-a1cd-2ad770aa292f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.613412 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.613265    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bf415666-9904-4e67-a1cd-2ad770aa292f-config-volume\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.614137 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.613903    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bf415666-9904-4e67-a1cd-2ad770aa292f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.614137 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.613909    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf415666-9904-4e67-a1cd-2ad770aa292f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.614967 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.614921    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf415666-9904-4e67-a1cd-2ad770aa292f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.616133 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.616102    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf415666-9904-4e67-a1cd-2ad770aa292f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.616254 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.616163    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bf415666-9904-4e67-a1cd-2ad770aa292f-web-config\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.616867 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.616589    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bf415666-9904-4e67-a1cd-2ad770aa292f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.616867 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.616803    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf415666-9904-4e67-a1cd-2ad770aa292f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.616867 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.616843    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bf415666-9904-4e67-a1cd-2ad770aa292f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.616867 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.616867    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bf415666-9904-4e67-a1cd-2ad770aa292f-config-volume\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.617114 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.617015    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf415666-9904-4e67-a1cd-2ad770aa292f-config-out\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.617194 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.617177    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bf415666-9904-4e67-a1cd-2ad770aa292f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.617444 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.617428    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bf415666-9904-4e67-a1cd-2ad770aa292f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.625642 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.625619    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfv8z\" (UniqueName: \"kubernetes.io/projected/bf415666-9904-4e67-a1cd-2ad770aa292f-kube-api-access-rfv8z\") pod \"alertmanager-main-0\" (UID: \"bf415666-9904-4e67-a1cd-2ad770aa292f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:23.719830 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:23.719792    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:24.012238 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:24.012182    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 20:53:24.014462 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:53:24.014429    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf415666_9904_4e67_a1cd_2ad770aa292f.slice/crio-a001d859d9aaa140cb16c4c028499f6903ee3f2c51ad445992d6be22398c5fba WatchSource:0}: Error finding container a001d859d9aaa140cb16c4c028499f6903ee3f2c51ad445992d6be22398c5fba: Status 404 returned error can't find the container with id a001d859d9aaa140cb16c4c028499f6903ee3f2c51ad445992d6be22398c5fba
Sep 17 20:53:24.355366 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:24.355276    2566 generic.go:356] "Generic (PLEG): container finished" podID="bf415666-9904-4e67-a1cd-2ad770aa292f" containerID="f170f9b0dc1e40e0c2910053a94c90cbde538bbc32f93f719c9f129ecae504a4" exitCode=0
Sep 17 20:53:24.355366 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:24.355336    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf415666-9904-4e67-a1cd-2ad770aa292f","Type":"ContainerDied","Data":"f170f9b0dc1e40e0c2910053a94c90cbde538bbc32f93f719c9f129ecae504a4"}
Sep 17 20:53:24.355588 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:24.355385    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf415666-9904-4e67-a1cd-2ad770aa292f","Type":"ContainerStarted","Data":"a001d859d9aaa140cb16c4c028499f6903ee3f2c51ad445992d6be22398c5fba"}
Sep 17 20:53:24.458840 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:24.458814    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91ea8785-e65b-4c0c-bbb7-e5514d627965" path="/var/lib/kubelet/pods/91ea8785-e65b-4c0c-bbb7-e5514d627965/volumes"
Sep 17 20:53:25.362639 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.362602    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf415666-9904-4e67-a1cd-2ad770aa292f","Type":"ContainerStarted","Data":"0a93c5b3c08fe825b40239d9c6decfd7ac1e74dfbf058db278f4d41393cfddd4"}
Sep 17 20:53:25.363055 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.362692    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf415666-9904-4e67-a1cd-2ad770aa292f","Type":"ContainerStarted","Data":"1e30ef700e6a561a1d0198bbf6f3e89b3b5d0a9a5a0402a3ab8094f391b658a0"}
Sep 17 20:53:25.363055 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.362718    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf415666-9904-4e67-a1cd-2ad770aa292f","Type":"ContainerStarted","Data":"85d30dba8516719209d97718bba41d2e5d2712ded271fe4e8040959dbf3dbd00"}
Sep 17 20:53:25.363055 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.362742    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf415666-9904-4e67-a1cd-2ad770aa292f","Type":"ContainerStarted","Data":"b6ef9327d37dda2b3d84d9fd57a5fcc42c2ab6bedaf82cc605fe460d3a797d58"}
Sep 17 20:53:25.363055 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.362766    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf415666-9904-4e67-a1cd-2ad770aa292f","Type":"ContainerStarted","Data":"0c155d1b45a0e296b4d126a115ec55019cbc8288916d6c605943667e7f040f45"}
Sep 17 20:53:25.363055 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.362787    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf415666-9904-4e67-a1cd-2ad770aa292f","Type":"ContainerStarted","Data":"88cd4ee727b275faa73361a4149ea9386c7adb1fafe052bc746886e6d1f86f0f"}
Sep 17 20:53:25.390897 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.390852    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.37613717 podStartE2EDuration="2.390840105s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.702935ms" isStatefulPod=false podCreationTimestamp="2026-09-17 20:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:53:25.389315469 +0000 UTC m=+247.000048702" watchObservedRunningTime="2026-09-17 20:53:25.390840105 +0000 UTC m=+247.001573348"
Sep 17 20:53:25.495646 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.495612    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2"]
Sep 17 20:53:25.499228 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.499190    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2"
Sep 17 20:53:25.502220 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.502186    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8wpjm\""
Sep 17 20:53:25.503028 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.502982    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 20:53:25.503249 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.503225    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 20:53:25.503356 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.503247    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 20:53:25.503428 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.503368    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 20:53:25.503564 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.503548    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 20:53:25.514110 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.514092    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 20:53:25.518419 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.518400    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2"]
Sep 17 20:53:25.630470 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.630372    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/566fd44b-5ea9-46f4-bbd1-86ed826cad04-telemeter-client-tls\") pod \"telemeter-client-55c8ddbbd9-2w5f2\" (UID: \"566fd44b-5ea9-46f4-bbd1-86ed826cad04\") " pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2"
Sep 17 20:53:25.630470 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.630406    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/566fd44b-5ea9-46f4-bbd1-86ed826cad04-metrics-client-ca\") pod \"telemeter-client-55c8ddbbd9-2w5f2\" (UID: \"566fd44b-5ea9-46f4-bbd1-86ed826cad04\") " pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2"
Sep 17 20:53:25.630470 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.630442    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/566fd44b-5ea9-46f4-bbd1-86ed826cad04-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55c8ddbbd9-2w5f2\" (UID: \"566fd44b-5ea9-46f4-bbd1-86ed826cad04\") " pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2"
Sep 17 20:53:25.630807 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.630558    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/566fd44b-5ea9-46f4-bbd1-86ed826cad04-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55c8ddbbd9-2w5f2\" (UID: \"566fd44b-5ea9-46f4-bbd1-86ed826cad04\") " pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2"
Sep 17 20:53:25.630807 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.630607    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/566fd44b-5ea9-46f4-bbd1-86ed826cad04-federate-client-tls\") pod \"telemeter-client-55c8ddbbd9-2w5f2\" (UID: \"566fd44b-5ea9-46f4-bbd1-86ed826cad04\") " pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2"
Sep 17 20:53:25.630807 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.630630    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/566fd44b-5ea9-46f4-bbd1-86ed826cad04-secret-telemeter-client\") pod \"telemeter-client-55c8ddbbd9-2w5f2\" (UID: \"566fd44b-5ea9-46f4-bbd1-86ed826cad04\") " pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2"
Sep 17 20:53:25.630807 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.630688    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/566fd44b-5ea9-46f4-bbd1-86ed826cad04-serving-certs-ca-bundle\") pod \"telemeter-client-55c8ddbbd9-2w5f2\" (UID: \"566fd44b-5ea9-46f4-bbd1-86ed826cad04\") " pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2"
Sep 17 20:53:25.630807 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.630723    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2fkg\" (UniqueName: \"kubernetes.io/projected/566fd44b-5ea9-46f4-bbd1-86ed826cad04-kube-api-access-l2fkg\") pod \"telemeter-client-55c8ddbbd9-2w5f2\" (UID: \"566fd44b-5ea9-46f4-bbd1-86ed826cad04\") " pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2"
Sep 17 20:53:25.731090 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.731058    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/566fd44b-5ea9-46f4-bbd1-86ed826cad04-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55c8ddbbd9-2w5f2\" (UID: \"566fd44b-5ea9-46f4-bbd1-86ed826cad04\") " pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2"
Sep 17 20:53:25.731293 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.731103    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/566fd44b-5ea9-46f4-bbd1-86ed826cad04-federate-client-tls\") pod \"telemeter-client-55c8ddbbd9-2w5f2\" (UID: \"566fd44b-5ea9-46f4-bbd1-86ed826cad04\") " pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2"
Sep 17 20:53:25.731293 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.731121    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/566fd44b-5ea9-46f4-bbd1-86ed826cad04-secret-telemeter-client\") pod \"telemeter-client-55c8ddbbd9-2w5f2\" (UID: \"566fd44b-5ea9-46f4-bbd1-86ed826cad04\") " pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2"
Sep 17 20:53:25.731293 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.731140    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/566fd44b-5ea9-46f4-bbd1-86ed826cad04-serving-certs-ca-bundle\") pod \"telemeter-client-55c8ddbbd9-2w5f2\" (UID: \"566fd44b-5ea9-46f4-bbd1-86ed826cad04\") " pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2"
Sep 17 20:53:25.731458 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.731293    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2fkg\" (UniqueName: \"kubernetes.io/projected/566fd44b-5ea9-46f4-bbd1-86ed826cad04-kube-api-access-l2fkg\") pod \"telemeter-client-55c8ddbbd9-2w5f2\" (UID: \"566fd44b-5ea9-46f4-bbd1-86ed826cad04\") " pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2"
Sep 17 20:53:25.731458 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.731352    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/566fd44b-5ea9-46f4-bbd1-86ed826cad04-telemeter-client-tls\") pod \"telemeter-client-55c8ddbbd9-2w5f2\" (UID: \"566fd44b-5ea9-46f4-bbd1-86ed826cad04\") " pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2"
Sep 17 20:53:25.731458 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.731399    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/566fd44b-5ea9-46f4-bbd1-86ed826cad04-metrics-client-ca\") pod \"telemeter-client-55c8ddbbd9-2w5f2\" (UID: \"566fd44b-5ea9-46f4-bbd1-86ed826cad04\") " pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2"
Sep 17 20:53:25.731603 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.731462    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/566fd44b-5ea9-46f4-bbd1-86ed826cad04-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55c8ddbbd9-2w5f2\" (UID: \"566fd44b-5ea9-46f4-bbd1-86ed826cad04\") " pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2"
Sep 17 20:53:25.731989 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.731967    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/566fd44b-5ea9-46f4-bbd1-86ed826cad04-serving-certs-ca-bundle\") pod \"telemeter-client-55c8ddbbd9-2w5f2\" (UID: \"566fd44b-5ea9-46f4-bbd1-86ed826cad04\") " pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2"
Sep 17 20:53:25.732075 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.732051    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/566fd44b-5ea9-46f4-bbd1-86ed826cad04-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55c8ddbbd9-2w5f2\" (UID: \"566fd44b-5ea9-46f4-bbd1-86ed826cad04\") " pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2"
Sep 17 20:53:25.732135 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.732098    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/566fd44b-5ea9-46f4-bbd1-86ed826cad04-metrics-client-ca\") pod \"telemeter-client-55c8ddbbd9-2w5f2\" (UID: \"566fd44b-5ea9-46f4-bbd1-86ed826cad04\") " pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2"
Sep 17 20:53:25.733784 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.733752    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/566fd44b-5ea9-46f4-bbd1-86ed826cad04-secret-telemeter-client\") pod \"telemeter-client-55c8ddbbd9-2w5f2\" (UID: \"566fd44b-5ea9-46f4-bbd1-86ed826cad04\") " pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2"
Sep 17 20:53:25.733784 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.733773    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/566fd44b-5ea9-46f4-bbd1-86ed826cad04-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55c8ddbbd9-2w5f2\" (UID: \"566fd44b-5ea9-46f4-bbd1-86ed826cad04\") " pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2"
Sep 17 20:53:25.733915 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.733761    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/566fd44b-5ea9-46f4-bbd1-86ed826cad04-telemeter-client-tls\") pod \"telemeter-client-55c8ddbbd9-2w5f2\" (UID: \"566fd44b-5ea9-46f4-bbd1-86ed826cad04\") " pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2"
Sep 17 20:53:25.733915 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.733841    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/566fd44b-5ea9-46f4-bbd1-86ed826cad04-federate-client-tls\") pod \"telemeter-client-55c8ddbbd9-2w5f2\" (UID: \"566fd44b-5ea9-46f4-bbd1-86ed826cad04\") " pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2"
Sep 17 20:53:25.739952 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.739930    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2fkg\" (UniqueName: \"kubernetes.io/projected/566fd44b-5ea9-46f4-bbd1-86ed826cad04-kube-api-access-l2fkg\") pod \"telemeter-client-55c8ddbbd9-2w5f2\" (UID: \"566fd44b-5ea9-46f4-bbd1-86ed826cad04\") " pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2"
Sep 17 20:53:25.809609 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:25.809577    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2"
Sep 17 20:53:26.085260 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:26.085054    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2"]
Sep 17 20:53:26.087951 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:53:26.087925    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod566fd44b_5ea9_46f4_bbd1_86ed826cad04.slice/crio-f79649673d6b46510f920d754b09e271c695411980db0bb19f5f3f2a014c3591 WatchSource:0}: Error finding container f79649673d6b46510f920d754b09e271c695411980db0bb19f5f3f2a014c3591: Status 404 returned error can't find the container with id f79649673d6b46510f920d754b09e271c695411980db0bb19f5f3f2a014c3591
Sep 17 20:53:26.368333 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:26.368243    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2" event={"ID":"566fd44b-5ea9-46f4-bbd1-86ed826cad04","Type":"ContainerStarted","Data":"f79649673d6b46510f920d754b09e271c695411980db0bb19f5f3f2a014c3591"}
Sep 17 20:53:28.378389 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:28.378356    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2" event={"ID":"566fd44b-5ea9-46f4-bbd1-86ed826cad04","Type":"ContainerStarted","Data":"d9c761c1e204ebe20569efad6fc5bdfd148b739ab0bad461ebe4a7f11d0564e5"}
Sep 17 20:53:28.378774 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:28.378448    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2" event={"ID":"566fd44b-5ea9-46f4-bbd1-86ed826cad04","Type":"ContainerStarted","Data":"3afca3048785b8ea4761c3da6c9c7bb2190697b1e847aaa5b2a05b8641b58a3e"}
Sep 17 20:53:28.378774 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:28.378469    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2" event={"ID":"566fd44b-5ea9-46f4-bbd1-86ed826cad04","Type":"ContainerStarted","Data":"64337dd4ab84e02080d697ebd897e35c45efe58323dba330b82407611580d451"}
Sep 17 20:53:28.402510 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:28.402464    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-55c8ddbbd9-2w5f2" podStartSLOduration=1.584699904 podStartE2EDuration="3.402448153s" totalImagesPullingTime="1.817748249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:53:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:53:28.402429297 +0000 UTC m=+250.013162527" watchObservedRunningTime="2026-09-17 20:53:28.402448153 +0000 UTC m=+250.013181364"
Sep 17 20:53:28.507486 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:28.507456    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75756cf756-tvg6m"]
Sep 17 20:53:28.510863 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:28.510832    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75756cf756-tvg6m"
Sep 17 20:53:28.526705 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:28.526677    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75756cf756-tvg6m"]
Sep 17 20:53:28.555883 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:28.555853    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h6mg\" (UniqueName: \"kubernetes.io/projected/a6aa3f07-fb21-43fd-a880-908528224369-kube-api-access-2h6mg\") pod \"console-75756cf756-tvg6m\" (UID: \"a6aa3f07-fb21-43fd-a880-908528224369\") " pod="openshift-console/console-75756cf756-tvg6m"
Sep 17 20:53:28.555883 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:28.555882    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6aa3f07-fb21-43fd-a880-908528224369-trusted-ca-bundle\") pod \"console-75756cf756-tvg6m\" (UID: \"a6aa3f07-fb21-43fd-a880-908528224369\") " pod="openshift-console/console-75756cf756-tvg6m"
Sep 17 20:53:28.556093 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:28.555904    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6aa3f07-fb21-43fd-a880-908528224369-oauth-serving-cert\") pod \"console-75756cf756-tvg6m\" (UID: \"a6aa3f07-fb21-43fd-a880-908528224369\") " pod="openshift-console/console-75756cf756-tvg6m"
Sep 17 20:53:28.556093 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:28.555974    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6aa3f07-fb21-43fd-a880-908528224369-service-ca\") pod \"console-75756cf756-tvg6m\" (UID: \"a6aa3f07-fb21-43fd-a880-908528224369\") " pod="openshift-console/console-75756cf756-tvg6m"
Sep 17 20:53:28.556093 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:28.556031    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6aa3f07-fb21-43fd-a880-908528224369-console-serving-cert\") pod \"console-75756cf756-tvg6m\" (UID: \"a6aa3f07-fb21-43fd-a880-908528224369\") " pod="openshift-console/console-75756cf756-tvg6m"
Sep 17 20:53:28.556093 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:28.556065    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6aa3f07-fb21-43fd-a880-908528224369-console-config\") pod \"console-75756cf756-tvg6m\" (UID: \"a6aa3f07-fb21-43fd-a880-908528224369\") " pod="openshift-console/console-75756cf756-tvg6m"
Sep 17 20:53:28.556271 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:28.556115    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6aa3f07-fb21-43fd-a880-908528224369-console-oauth-config\") pod \"console-75756cf756-tvg6m\" (UID: \"a6aa3f07-fb21-43fd-a880-908528224369\") " pod="openshift-console/console-75756cf756-tvg6m"
Sep 17 20:53:28.656636 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:28.656548    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h6mg\" (UniqueName: \"kubernetes.io/projected/a6aa3f07-fb21-43fd-a880-908528224369-kube-api-access-2h6mg\") pod \"console-75756cf756-tvg6m\" (UID: \"a6aa3f07-fb21-43fd-a880-908528224369\") " pod="openshift-console/console-75756cf756-tvg6m"
Sep 17 20:53:28.656636 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:28.656583    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6aa3f07-fb21-43fd-a880-908528224369-trusted-ca-bundle\") pod \"console-75756cf756-tvg6m\" (UID: \"a6aa3f07-fb21-43fd-a880-908528224369\") " pod="openshift-console/console-75756cf756-tvg6m"
Sep 17 20:53:28.656636 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:28.656603    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6aa3f07-fb21-43fd-a880-908528224369-oauth-serving-cert\") pod \"console-75756cf756-tvg6m\" (UID: \"a6aa3f07-fb21-43fd-a880-908528224369\") " pod="openshift-console/console-75756cf756-tvg6m"
Sep 17 20:53:28.656888 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:28.656640    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6aa3f07-fb21-43fd-a880-908528224369-service-ca\") pod \"console-75756cf756-tvg6m\" (UID: \"a6aa3f07-fb21-43fd-a880-908528224369\") " pod="openshift-console/console-75756cf756-tvg6m"
Sep 17 20:53:28.656888 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:28.656664    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6aa3f07-fb21-43fd-a880-908528224369-console-serving-cert\") pod \"console-75756cf756-tvg6m\" (UID: \"a6aa3f07-fb21-43fd-a880-908528224369\") " pod="openshift-console/console-75756cf756-tvg6m"
Sep 17 20:53:28.656888 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:28.656678    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6aa3f07-fb21-43fd-a880-908528224369-console-config\") pod \"console-75756cf756-tvg6m\" (UID: \"a6aa3f07-fb21-43fd-a880-908528224369\") " pod="openshift-console/console-75756cf756-tvg6m"
Sep 17 20:53:28.656888 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:28.656699    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6aa3f07-fb21-43fd-a880-908528224369-console-oauth-config\") pod \"console-75756cf756-tvg6m\" (UID: \"a6aa3f07-fb21-43fd-a880-908528224369\") " pod="openshift-console/console-75756cf756-tvg6m"
Sep 17 20:53:28.657484 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:28.657457    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6aa3f07-fb21-43fd-a880-908528224369-service-ca\") pod \"console-75756cf756-tvg6m\" (UID: \"a6aa3f07-fb21-43fd-a880-908528224369\") " pod="openshift-console/console-75756cf756-tvg6m"
Sep 17 20:53:28.657623 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:28.657495    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6aa3f07-fb21-43fd-a880-908528224369-trusted-ca-bundle\") pod \"console-75756cf756-tvg6m\" (UID: \"a6aa3f07-fb21-43fd-a880-908528224369\") " pod="openshift-console/console-75756cf756-tvg6m"
Sep 17 20:53:28.657623 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:28.657515    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6aa3f07-fb21-43fd-a880-908528224369-console-config\") pod \"console-75756cf756-tvg6m\" (UID: \"a6aa3f07-fb21-43fd-a880-908528224369\") " pod="openshift-console/console-75756cf756-tvg6m"
Sep 17 20:53:28.657623 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:28.657562    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6aa3f07-fb21-43fd-a880-908528224369-oauth-serving-cert\") pod \"console-75756cf756-tvg6m\" (UID: \"a6aa3f07-fb21-43fd-a880-908528224369\") " pod="openshift-console/console-75756cf756-tvg6m"
Sep 17 20:53:28.659243 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:28.659189    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6aa3f07-fb21-43fd-a880-908528224369-console-serving-cert\") pod \"console-75756cf756-tvg6m\" (UID: \"a6aa3f07-fb21-43fd-a880-908528224369\") " pod="openshift-console/console-75756cf756-tvg6m"
Sep 17 20:53:28.659339 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:28.659261    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6aa3f07-fb21-43fd-a880-908528224369-console-oauth-config\") pod \"console-75756cf756-tvg6m\" (UID: \"a6aa3f07-fb21-43fd-a880-908528224369\") " pod="openshift-console/console-75756cf756-tvg6m"
Sep 17 20:53:28.665986 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:28.665965    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h6mg\" (UniqueName: \"kubernetes.io/projected/a6aa3f07-fb21-43fd-a880-908528224369-kube-api-access-2h6mg\") pod \"console-75756cf756-tvg6m\" (UID: \"a6aa3f07-fb21-43fd-a880-908528224369\") " pod="openshift-console/console-75756cf756-tvg6m"
Sep 17 20:53:28.819835 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:28.819803    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75756cf756-tvg6m"
Sep 17 20:53:29.090194 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:29.090065    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75756cf756-tvg6m"]
Sep 17 20:53:29.092606 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:53:29.092580    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6aa3f07_fb21_43fd_a880_908528224369.slice/crio-d774ada331d08aa7201577976043701079b289a47d89c1e88e6c0f434d8db873 WatchSource:0}: Error finding container d774ada331d08aa7201577976043701079b289a47d89c1e88e6c0f434d8db873: Status 404 returned error can't find the container with id d774ada331d08aa7201577976043701079b289a47d89c1e88e6c0f434d8db873
Sep 17 20:53:29.383477 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:29.383381    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75756cf756-tvg6m" event={"ID":"a6aa3f07-fb21-43fd-a880-908528224369","Type":"ContainerStarted","Data":"1eecfea305fa7df219d34459f0e9c7a3fc00819e31f0e9e0569097db7fafba25"}
Sep 17 20:53:29.383477 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:29.383432    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75756cf756-tvg6m" event={"ID":"a6aa3f07-fb21-43fd-a880-908528224369","Type":"ContainerStarted","Data":"d774ada331d08aa7201577976043701079b289a47d89c1e88e6c0f434d8db873"}
Sep 17 20:53:29.423445 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:29.421933    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75756cf756-tvg6m" podStartSLOduration=1.4219140129999999 podStartE2EDuration="1.421914013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:53:29.420803691 +0000 UTC m=+251.031536920" watchObservedRunningTime="2026-09-17 20:53:29.421914013 +0000 UTC m=+251.032647245"
Sep 17 20:53:32.183271 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:32.183204    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-metrics-certs\") pod \"network-metrics-daemon-sknnf\" (UID: \"8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7\") " pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:53:32.185476 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:32.185448    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7-metrics-certs\") pod \"network-metrics-daemon-sknnf\" (UID: \"8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7\") " pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:53:32.359345 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:32.359312    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6gfd5\""
Sep 17 20:53:32.366801 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:32.366782    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sknnf"
Sep 17 20:53:32.635828 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:32.635795    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sknnf"]
Sep 17 20:53:32.638693 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:53:32.638671    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d4efba7_875e_4c04_9ebe_2bd9e0e0dfa7.slice/crio-7e6d899c70ee0394204ce9f6478557409b2613747cbf4f9f79530d5926065f6a WatchSource:0}: Error finding container 7e6d899c70ee0394204ce9f6478557409b2613747cbf4f9f79530d5926065f6a: Status 404 returned error can't find the container with id 7e6d899c70ee0394204ce9f6478557409b2613747cbf4f9f79530d5926065f6a
Sep 17 20:53:33.399700 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:33.399605    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sknnf" event={"ID":"8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7","Type":"ContainerStarted","Data":"7e6d899c70ee0394204ce9f6478557409b2613747cbf4f9f79530d5926065f6a"}
Sep 17 20:53:34.404971 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:34.404931    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sknnf" event={"ID":"8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7","Type":"ContainerStarted","Data":"cb23c98b550bc6d4d7e89939890508af8a532093e834a948252f596a674a48c5"}
Sep 17 20:53:34.405396 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:34.404991    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sknnf" event={"ID":"8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7","Type":"ContainerStarted","Data":"4c1d1beaf9a69e7297bc010eb5b88aa754231141e0784bdc122f6ad59f428a56"}
Sep 17 20:53:34.422363 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:34.422317    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sknnf" podStartSLOduration=253.4637664 podStartE2EDuration="4m14.422301655s" totalImagesPullingTime="958.535255ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:53:34.421582838 +0000 UTC m=+256.032316069" watchObservedRunningTime="2026-09-17 20:53:34.422301655 +0000 UTC m=+256.033034888"
Sep 17 20:53:38.820609 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:38.820570    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75756cf756-tvg6m"
Sep 17 20:53:38.820962 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:38.820641    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75756cf756-tvg6m"
Sep 17 20:53:38.825174 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:38.825157    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75756cf756-tvg6m"
Sep 17 20:53:39.425386 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:39.425358    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75756cf756-tvg6m"
Sep 17 20:53:39.475449 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:53:39.475420    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66c8bb9b49-bd9vr"]
Sep 17 20:54:04.494985 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:04.494930    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66c8bb9b49-bd9vr" podUID="34006e50-f738-41b0-b6e4-055e6694bad5" containerName="console" containerID="cri-o://83cfb166d9944df45d1a8da44cf4f0f14b751af0d1e802897e8bded31a9f1664" gracePeriod=15
Sep 17 20:54:04.736116 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:04.736092    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66c8bb9b49-bd9vr"
Sep 17 20:54:04.925653 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:04.925562    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34006e50-f738-41b0-b6e4-055e6694bad5-trusted-ca-bundle\") pod \"34006e50-f738-41b0-b6e4-055e6694bad5\" (UID: \"34006e50-f738-41b0-b6e4-055e6694bad5\") "
Sep 17 20:54:04.925653 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:04.925617    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/34006e50-f738-41b0-b6e4-055e6694bad5-service-ca\") pod \"34006e50-f738-41b0-b6e4-055e6694bad5\" (UID: \"34006e50-f738-41b0-b6e4-055e6694bad5\") "
Sep 17 20:54:04.925653 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:04.925641    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8tl5\" (UniqueName: \"kubernetes.io/projected/34006e50-f738-41b0-b6e4-055e6694bad5-kube-api-access-b8tl5\") pod \"34006e50-f738-41b0-b6e4-055e6694bad5\" (UID: \"34006e50-f738-41b0-b6e4-055e6694bad5\") "
Sep 17 20:54:04.925937 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:04.925667    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34006e50-f738-41b0-b6e4-055e6694bad5-console-oauth-config\") pod \"34006e50-f738-41b0-b6e4-055e6694bad5\" (UID: \"34006e50-f738-41b0-b6e4-055e6694bad5\") "
Sep 17 20:54:04.925937 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:04.925691    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34006e50-f738-41b0-b6e4-055e6694bad5-console-serving-cert\") pod \"34006e50-f738-41b0-b6e4-055e6694bad5\" (UID: \"34006e50-f738-41b0-b6e4-055e6694bad5\") "
Sep 17 20:54:04.925937 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:04.925716    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/34006e50-f738-41b0-b6e4-055e6694bad5-console-config\") pod \"34006e50-f738-41b0-b6e4-055e6694bad5\" (UID: \"34006e50-f738-41b0-b6e4-055e6694bad5\") "
Sep 17 20:54:04.925937 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:04.925734    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34006e50-f738-41b0-b6e4-055e6694bad5-oauth-serving-cert\") pod \"34006e50-f738-41b0-b6e4-055e6694bad5\" (UID: \"34006e50-f738-41b0-b6e4-055e6694bad5\") "
Sep 17 20:54:04.926117 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:04.925981    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34006e50-f738-41b0-b6e4-055e6694bad5-service-ca" (OuterVolumeSpecName: "service-ca") pod "34006e50-f738-41b0-b6e4-055e6694bad5" (UID: "34006e50-f738-41b0-b6e4-055e6694bad5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:54:04.926117 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:04.926017    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34006e50-f738-41b0-b6e4-055e6694bad5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "34006e50-f738-41b0-b6e4-055e6694bad5" (UID: "34006e50-f738-41b0-b6e4-055e6694bad5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:54:04.926229 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:04.926161    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34006e50-f738-41b0-b6e4-055e6694bad5-console-config" (OuterVolumeSpecName: "console-config") pod "34006e50-f738-41b0-b6e4-055e6694bad5" (UID: "34006e50-f738-41b0-b6e4-055e6694bad5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:54:04.926281 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:04.926240    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34006e50-f738-41b0-b6e4-055e6694bad5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "34006e50-f738-41b0-b6e4-055e6694bad5" (UID: "34006e50-f738-41b0-b6e4-055e6694bad5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:54:04.927912 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:04.927881    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34006e50-f738-41b0-b6e4-055e6694bad5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "34006e50-f738-41b0-b6e4-055e6694bad5" (UID: "34006e50-f738-41b0-b6e4-055e6694bad5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:54:04.928017 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:04.927931    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34006e50-f738-41b0-b6e4-055e6694bad5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "34006e50-f738-41b0-b6e4-055e6694bad5" (UID: "34006e50-f738-41b0-b6e4-055e6694bad5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:54:04.928017 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:04.927934    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34006e50-f738-41b0-b6e4-055e6694bad5-kube-api-access-b8tl5" (OuterVolumeSpecName: "kube-api-access-b8tl5") pod "34006e50-f738-41b0-b6e4-055e6694bad5" (UID: "34006e50-f738-41b0-b6e4-055e6694bad5"). InnerVolumeSpecName "kube-api-access-b8tl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:54:05.026873 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:05.026839    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34006e50-f738-41b0-b6e4-055e6694bad5-console-oauth-config\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:54:05.026873 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:05.026869    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34006e50-f738-41b0-b6e4-055e6694bad5-console-serving-cert\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:54:05.026873 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:05.026879    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/34006e50-f738-41b0-b6e4-055e6694bad5-console-config\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:54:05.027051 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:05.026888    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34006e50-f738-41b0-b6e4-055e6694bad5-oauth-serving-cert\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:54:05.027051 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:05.026896    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34006e50-f738-41b0-b6e4-055e6694bad5-trusted-ca-bundle\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:54:05.027051 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:05.026904    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/34006e50-f738-41b0-b6e4-055e6694bad5-service-ca\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:54:05.027051 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:05.026912    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8tl5\" (UniqueName: \"kubernetes.io/projected/34006e50-f738-41b0-b6e4-055e6694bad5-kube-api-access-b8tl5\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:54:05.523259 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:05.523228    2566 generic.go:356] "Generic (PLEG): container finished" podID="34006e50-f738-41b0-b6e4-055e6694bad5" containerID="83cfb166d9944df45d1a8da44cf4f0f14b751af0d1e802897e8bded31a9f1664" exitCode=0
Sep 17 20:54:05.523693 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:05.523270    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66c8bb9b49-bd9vr"
Sep 17 20:54:05.523693 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:05.523291    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66c8bb9b49-bd9vr" event={"ID":"34006e50-f738-41b0-b6e4-055e6694bad5","Type":"ContainerDied","Data":"83cfb166d9944df45d1a8da44cf4f0f14b751af0d1e802897e8bded31a9f1664"}
Sep 17 20:54:05.523693 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:05.523330    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66c8bb9b49-bd9vr" event={"ID":"34006e50-f738-41b0-b6e4-055e6694bad5","Type":"ContainerDied","Data":"8c720dc08057a06d0c35e25b8ee756119ea32c222642150cc62671d35a24508d"}
Sep 17 20:54:05.523693 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:05.523349    2566 scope.go:117] "RemoveContainer" containerID="83cfb166d9944df45d1a8da44cf4f0f14b751af0d1e802897e8bded31a9f1664"
Sep 17 20:54:05.531754 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:05.531739    2566 scope.go:117] "RemoveContainer" containerID="83cfb166d9944df45d1a8da44cf4f0f14b751af0d1e802897e8bded31a9f1664"
Sep 17 20:54:05.532017 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:54:05.531985    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83cfb166d9944df45d1a8da44cf4f0f14b751af0d1e802897e8bded31a9f1664\": container with ID starting with 83cfb166d9944df45d1a8da44cf4f0f14b751af0d1e802897e8bded31a9f1664 not found: ID does not exist" containerID="83cfb166d9944df45d1a8da44cf4f0f14b751af0d1e802897e8bded31a9f1664"
Sep 17 20:54:05.532052 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:05.532026    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83cfb166d9944df45d1a8da44cf4f0f14b751af0d1e802897e8bded31a9f1664"} err="failed to get container status \"83cfb166d9944df45d1a8da44cf4f0f14b751af0d1e802897e8bded31a9f1664\": rpc error: code = NotFound desc = could not find container \"83cfb166d9944df45d1a8da44cf4f0f14b751af0d1e802897e8bded31a9f1664\": container with ID starting with 83cfb166d9944df45d1a8da44cf4f0f14b751af0d1e802897e8bded31a9f1664 not found: ID does not exist"
Sep 17 20:54:05.548324 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:05.548298    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66c8bb9b49-bd9vr"]
Sep 17 20:54:05.549372 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:05.549355    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66c8bb9b49-bd9vr"]
Sep 17 20:54:06.458700 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:06.458666    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34006e50-f738-41b0-b6e4-055e6694bad5" path="/var/lib/kubelet/pods/34006e50-f738-41b0-b6e4-055e6694bad5/volumes"
Sep 17 20:54:20.357243 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:20.357200    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t2rlv_ed6472a6-7979-43b0-a549-7ca5c5d8fb0f/console-operator/1.log"
Sep 17 20:54:20.359381 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:20.359359    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t2rlv_ed6472a6-7979-43b0-a549-7ca5c5d8fb0f/console-operator/1.log"
Sep 17 20:54:20.362518 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:20.362479    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnqx7_402f55a4-24b6-47c6-b719-9c6a5b473434/ovn-acl-logging/0.log"
Sep 17 20:54:20.364641 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:20.364623    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnqx7_402f55a4-24b6-47c6-b719-9c6a5b473434/ovn-acl-logging/0.log"
Sep 17 20:54:20.369713 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:20.369695    2566 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 20:54:37.079237 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:37.079190    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-694ff8fcb5-jscsd"]
Sep 17 20:54:37.081670 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:37.079472    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34006e50-f738-41b0-b6e4-055e6694bad5" containerName="console"
Sep 17 20:54:37.081670 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:37.079483    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="34006e50-f738-41b0-b6e4-055e6694bad5" containerName="console"
Sep 17 20:54:37.081670 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:37.079542    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="34006e50-f738-41b0-b6e4-055e6694bad5" containerName="console"
Sep 17 20:54:37.082536 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:37.082486    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-694ff8fcb5-jscsd"
Sep 17 20:54:37.096780 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:37.096757    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-694ff8fcb5-jscsd"]
Sep 17 20:54:37.142706 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:37.142674    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-service-ca\") pod \"console-694ff8fcb5-jscsd\" (UID: \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\") " pod="openshift-console/console-694ff8fcb5-jscsd"
Sep 17 20:54:37.142706 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:37.142703    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-oauth-serving-cert\") pod \"console-694ff8fcb5-jscsd\" (UID: \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\") " pod="openshift-console/console-694ff8fcb5-jscsd"
Sep 17 20:54:37.142873 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:37.142753    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-trusted-ca-bundle\") pod \"console-694ff8fcb5-jscsd\" (UID: \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\") " pod="openshift-console/console-694ff8fcb5-jscsd"
Sep 17 20:54:37.142873 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:37.142777    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-console-config\") pod \"console-694ff8fcb5-jscsd\" (UID: \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\") " pod="openshift-console/console-694ff8fcb5-jscsd"
Sep 17 20:54:37.142873 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:37.142846    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkxpc\" (UniqueName: \"kubernetes.io/projected/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-kube-api-access-tkxpc\") pod \"console-694ff8fcb5-jscsd\" (UID: \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\") " pod="openshift-console/console-694ff8fcb5-jscsd"
Sep 17 20:54:37.142968 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:37.142878    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-console-oauth-config\") pod \"console-694ff8fcb5-jscsd\" (UID: \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\") " pod="openshift-console/console-694ff8fcb5-jscsd"
Sep 17 20:54:37.142968 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:37.142899    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-console-serving-cert\") pod \"console-694ff8fcb5-jscsd\" (UID: \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\") " pod="openshift-console/console-694ff8fcb5-jscsd"
Sep 17 20:54:37.243598 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:37.243568    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-trusted-ca-bundle\") pod \"console-694ff8fcb5-jscsd\" (UID: \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\") " pod="openshift-console/console-694ff8fcb5-jscsd"
Sep 17 20:54:37.243722 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:37.243610    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-console-config\") pod \"console-694ff8fcb5-jscsd\" (UID: \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\") " pod="openshift-console/console-694ff8fcb5-jscsd"
Sep 17 20:54:37.243722 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:37.243652    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkxpc\" (UniqueName: \"kubernetes.io/projected/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-kube-api-access-tkxpc\") pod \"console-694ff8fcb5-jscsd\" (UID: \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\") " pod="openshift-console/console-694ff8fcb5-jscsd"
Sep 17 20:54:37.243722 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:37.243692    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-console-oauth-config\") pod \"console-694ff8fcb5-jscsd\" (UID: \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\") " pod="openshift-console/console-694ff8fcb5-jscsd"
Sep 17 20:54:37.243820 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:37.243728    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-console-serving-cert\") pod \"console-694ff8fcb5-jscsd\" (UID: \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\") " pod="openshift-console/console-694ff8fcb5-jscsd"
Sep 17 20:54:37.243820 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:37.243777    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-service-ca\") pod \"console-694ff8fcb5-jscsd\" (UID: \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\") " pod="openshift-console/console-694ff8fcb5-jscsd"
Sep 17 20:54:37.243820 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:37.243798    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-oauth-serving-cert\") pod \"console-694ff8fcb5-jscsd\" (UID: \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\") " pod="openshift-console/console-694ff8fcb5-jscsd"
Sep 17 20:54:37.244482 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:37.244451    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-service-ca\") pod \"console-694ff8fcb5-jscsd\" (UID: \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\") " pod="openshift-console/console-694ff8fcb5-jscsd"
Sep 17 20:54:37.244574 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:37.244453    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-console-config\") pod \"console-694ff8fcb5-jscsd\" (UID: \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\") " pod="openshift-console/console-694ff8fcb5-jscsd"
Sep 17 20:54:37.244574 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:37.244504    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-oauth-serving-cert\") pod \"console-694ff8fcb5-jscsd\" (UID: \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\") " pod="openshift-console/console-694ff8fcb5-jscsd"
Sep 17 20:54:37.244574 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:37.244558    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-trusted-ca-bundle\") pod \"console-694ff8fcb5-jscsd\" (UID: \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\") " pod="openshift-console/console-694ff8fcb5-jscsd"
Sep 17 20:54:37.246055 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:37.246033    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-console-oauth-config\") pod \"console-694ff8fcb5-jscsd\" (UID: \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\") " pod="openshift-console/console-694ff8fcb5-jscsd"
Sep 17 20:54:37.246163 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:37.246117    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-console-serving-cert\") pod \"console-694ff8fcb5-jscsd\" (UID: \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\") " pod="openshift-console/console-694ff8fcb5-jscsd"
Sep 17 20:54:37.253106 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:37.253086    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkxpc\" (UniqueName: \"kubernetes.io/projected/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-kube-api-access-tkxpc\") pod \"console-694ff8fcb5-jscsd\" (UID: \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\") " pod="openshift-console/console-694ff8fcb5-jscsd"
Sep 17 20:54:37.392006 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:37.391931    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-694ff8fcb5-jscsd"
Sep 17 20:54:37.661515 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:37.661493    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-694ff8fcb5-jscsd"]
Sep 17 20:54:37.663870 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:54:37.663841    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc347dc67_05b9_4b92_9609_bc8a09b9ad7e.slice/crio-f04ee81d0a2289d5b9a1dd29644fdc4ce0dc1e9f6e64d50fa7aa6fe7e921fb5f WatchSource:0}: Error finding container f04ee81d0a2289d5b9a1dd29644fdc4ce0dc1e9f6e64d50fa7aa6fe7e921fb5f: Status 404 returned error can't find the container with id f04ee81d0a2289d5b9a1dd29644fdc4ce0dc1e9f6e64d50fa7aa6fe7e921fb5f
Sep 17 20:54:38.641162 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:38.641125    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-694ff8fcb5-jscsd" event={"ID":"c347dc67-05b9-4b92-9609-bc8a09b9ad7e","Type":"ContainerStarted","Data":"b128501103b7dfe3eb35fde64dfd2f975cca22cb662b386deadf131e62e842d7"}
Sep 17 20:54:38.641162 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:38.641169    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-694ff8fcb5-jscsd" event={"ID":"c347dc67-05b9-4b92-9609-bc8a09b9ad7e","Type":"ContainerStarted","Data":"f04ee81d0a2289d5b9a1dd29644fdc4ce0dc1e9f6e64d50fa7aa6fe7e921fb5f"}
Sep 17 20:54:38.665991 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:38.665940    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-694ff8fcb5-jscsd" podStartSLOduration=1.665926888 podStartE2EDuration="1.665926888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:54:38.664688716 +0000 UTC m=+320.275421946" watchObservedRunningTime="2026-09-17 20:54:38.665926888 +0000 UTC m=+320.276660117"
Sep 17 20:54:47.392931 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:47.392897    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-694ff8fcb5-jscsd"
Sep 17 20:54:47.393400 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:47.392970    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-694ff8fcb5-jscsd"
Sep 17 20:54:47.397466 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:47.397445    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-694ff8fcb5-jscsd"
Sep 17 20:54:47.676363 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:47.676296    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-694ff8fcb5-jscsd"
Sep 17 20:54:47.725683 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:54:47.725650    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75756cf756-tvg6m"]
Sep 17 20:55:12.746877 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:12.746760    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75756cf756-tvg6m" podUID="a6aa3f07-fb21-43fd-a880-908528224369" containerName="console" containerID="cri-o://1eecfea305fa7df219d34459f0e9c7a3fc00819e31f0e9e0569097db7fafba25" gracePeriod=15
Sep 17 20:55:12.980837 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:12.980812    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75756cf756-tvg6m"
Sep 17 20:55:13.122706 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:13.122625    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6aa3f07-fb21-43fd-a880-908528224369-oauth-serving-cert\") pod \"a6aa3f07-fb21-43fd-a880-908528224369\" (UID: \"a6aa3f07-fb21-43fd-a880-908528224369\") "
Sep 17 20:55:13.122706 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:13.122663    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h6mg\" (UniqueName: \"kubernetes.io/projected/a6aa3f07-fb21-43fd-a880-908528224369-kube-api-access-2h6mg\") pod \"a6aa3f07-fb21-43fd-a880-908528224369\" (UID: \"a6aa3f07-fb21-43fd-a880-908528224369\") "
Sep 17 20:55:13.122706 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:13.122679    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6aa3f07-fb21-43fd-a880-908528224369-console-oauth-config\") pod \"a6aa3f07-fb21-43fd-a880-908528224369\" (UID: \"a6aa3f07-fb21-43fd-a880-908528224369\") "
Sep 17 20:55:13.122965 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:13.122721    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6aa3f07-fb21-43fd-a880-908528224369-service-ca\") pod \"a6aa3f07-fb21-43fd-a880-908528224369\" (UID: \"a6aa3f07-fb21-43fd-a880-908528224369\") "
Sep 17 20:55:13.122965 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:13.122752    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6aa3f07-fb21-43fd-a880-908528224369-console-config\") pod \"a6aa3f07-fb21-43fd-a880-908528224369\" (UID: \"a6aa3f07-fb21-43fd-a880-908528224369\") "
Sep 17 20:55:13.122965 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:13.122769    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6aa3f07-fb21-43fd-a880-908528224369-trusted-ca-bundle\") pod \"a6aa3f07-fb21-43fd-a880-908528224369\" (UID: \"a6aa3f07-fb21-43fd-a880-908528224369\") "
Sep 17 20:55:13.122965 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:13.122789    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6aa3f07-fb21-43fd-a880-908528224369-console-serving-cert\") pod \"a6aa3f07-fb21-43fd-a880-908528224369\" (UID: \"a6aa3f07-fb21-43fd-a880-908528224369\") "
Sep 17 20:55:13.123169 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:13.123126    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6aa3f07-fb21-43fd-a880-908528224369-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a6aa3f07-fb21-43fd-a880-908528224369" (UID: "a6aa3f07-fb21-43fd-a880-908528224369"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:55:13.123169 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:13.123133    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6aa3f07-fb21-43fd-a880-908528224369-service-ca" (OuterVolumeSpecName: "service-ca") pod "a6aa3f07-fb21-43fd-a880-908528224369" (UID: "a6aa3f07-fb21-43fd-a880-908528224369"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:55:13.123322 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:13.123303    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6aa3f07-fb21-43fd-a880-908528224369-console-config" (OuterVolumeSpecName: "console-config") pod "a6aa3f07-fb21-43fd-a880-908528224369" (UID: "a6aa3f07-fb21-43fd-a880-908528224369"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:55:13.123377 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:13.123359    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6aa3f07-fb21-43fd-a880-908528224369-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a6aa3f07-fb21-43fd-a880-908528224369" (UID: "a6aa3f07-fb21-43fd-a880-908528224369"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:55:13.124841 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:13.124822    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6aa3f07-fb21-43fd-a880-908528224369-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a6aa3f07-fb21-43fd-a880-908528224369" (UID: "a6aa3f07-fb21-43fd-a880-908528224369"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:55:13.125383 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:13.125361    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6aa3f07-fb21-43fd-a880-908528224369-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a6aa3f07-fb21-43fd-a880-908528224369" (UID: "a6aa3f07-fb21-43fd-a880-908528224369"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:55:13.125436 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:13.125361    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6aa3f07-fb21-43fd-a880-908528224369-kube-api-access-2h6mg" (OuterVolumeSpecName: "kube-api-access-2h6mg") pod "a6aa3f07-fb21-43fd-a880-908528224369" (UID: "a6aa3f07-fb21-43fd-a880-908528224369"). InnerVolumeSpecName "kube-api-access-2h6mg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:55:13.223895 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:13.223866    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6aa3f07-fb21-43fd-a880-908528224369-service-ca\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:55:13.223895 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:13.223889    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6aa3f07-fb21-43fd-a880-908528224369-console-config\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:55:13.223895 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:13.223898    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6aa3f07-fb21-43fd-a880-908528224369-trusted-ca-bundle\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:55:13.224084 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:13.223907    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6aa3f07-fb21-43fd-a880-908528224369-console-serving-cert\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:55:13.224084 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:13.223915    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6aa3f07-fb21-43fd-a880-908528224369-oauth-serving-cert\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:55:13.224084 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:13.223924    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h6mg\" (UniqueName: \"kubernetes.io/projected/a6aa3f07-fb21-43fd-a880-908528224369-kube-api-access-2h6mg\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:55:13.224084 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:13.223932    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6aa3f07-fb21-43fd-a880-908528224369-console-oauth-config\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:55:13.766764 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:13.766729    2566 generic.go:356] "Generic (PLEG): container finished" podID="a6aa3f07-fb21-43fd-a880-908528224369" containerID="1eecfea305fa7df219d34459f0e9c7a3fc00819e31f0e9e0569097db7fafba25" exitCode=0
Sep 17 20:55:13.766764 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:13.766769    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75756cf756-tvg6m"
Sep 17 20:55:13.767291 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:13.766788    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75756cf756-tvg6m" event={"ID":"a6aa3f07-fb21-43fd-a880-908528224369","Type":"ContainerDied","Data":"1eecfea305fa7df219d34459f0e9c7a3fc00819e31f0e9e0569097db7fafba25"}
Sep 17 20:55:13.767291 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:13.766835    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75756cf756-tvg6m" event={"ID":"a6aa3f07-fb21-43fd-a880-908528224369","Type":"ContainerDied","Data":"d774ada331d08aa7201577976043701079b289a47d89c1e88e6c0f434d8db873"}
Sep 17 20:55:13.767291 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:13.766863    2566 scope.go:117] "RemoveContainer" containerID="1eecfea305fa7df219d34459f0e9c7a3fc00819e31f0e9e0569097db7fafba25"
Sep 17 20:55:13.775401 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:13.775382    2566 scope.go:117] "RemoveContainer" containerID="1eecfea305fa7df219d34459f0e9c7a3fc00819e31f0e9e0569097db7fafba25"
Sep 17 20:55:13.775653 ip-10-0-138-140 kubenswrapper[2566]: E0917 20:55:13.775635    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eecfea305fa7df219d34459f0e9c7a3fc00819e31f0e9e0569097db7fafba25\": container with ID starting with 1eecfea305fa7df219d34459f0e9c7a3fc00819e31f0e9e0569097db7fafba25 not found: ID does not exist" containerID="1eecfea305fa7df219d34459f0e9c7a3fc00819e31f0e9e0569097db7fafba25"
Sep 17 20:55:13.775706 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:13.775660    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eecfea305fa7df219d34459f0e9c7a3fc00819e31f0e9e0569097db7fafba25"} err="failed to get container status \"1eecfea305fa7df219d34459f0e9c7a3fc00819e31f0e9e0569097db7fafba25\": rpc error: code = NotFound desc = could not find container \"1eecfea305fa7df219d34459f0e9c7a3fc00819e31f0e9e0569097db7fafba25\": container with ID starting with 1eecfea305fa7df219d34459f0e9c7a3fc00819e31f0e9e0569097db7fafba25 not found: ID does not exist"
Sep 17 20:55:13.786959 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:13.786925    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75756cf756-tvg6m"]
Sep 17 20:55:13.788641 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:13.788622    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75756cf756-tvg6m"]
Sep 17 20:55:14.458900 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:14.458866    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6aa3f07-fb21-43fd-a880-908528224369" path="/var/lib/kubelet/pods/a6aa3f07-fb21-43fd-a880-908528224369/volumes"
Sep 17 20:55:42.244919 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.244888    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c6c7465c4-2d8kq"]
Sep 17 20:55:42.245401 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.245181    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6aa3f07-fb21-43fd-a880-908528224369" containerName="console"
Sep 17 20:55:42.245401 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.245190    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6aa3f07-fb21-43fd-a880-908528224369" containerName="console"
Sep 17 20:55:42.245401 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.245256    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6aa3f07-fb21-43fd-a880-908528224369" containerName="console"
Sep 17 20:55:42.248022 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.248007    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c6c7465c4-2d8kq"
Sep 17 20:55:42.250696 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.250672    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 20:55:42.251053 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.251034    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 20:55:42.251159 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.251141    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 20:55:42.252110 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.252094    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-7fzqf\""
Sep 17 20:55:42.252204 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.252154    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 20:55:42.261524 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.261502    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c6c7465c4-2d8kq"]
Sep 17 20:55:42.310851 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.310823    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cdccc466-x6w9n"]
Sep 17 20:55:42.313990 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.313972    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cdccc466-x6w9n"
Sep 17 20:55:42.316772 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.316752    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 20:55:42.316859 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.316768    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 20:55:42.316859 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.316758    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 20:55:42.316859 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.316817    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 20:55:42.330785 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.330760    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cdccc466-x6w9n"]
Sep 17 20:55:42.332551 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.332521    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/49d98dab-dc56-41d3-b85b-758096e539cc-hub\") pod \"cluster-proxy-proxy-agent-76cdccc466-x6w9n\" (UID: \"49d98dab-dc56-41d3-b85b-758096e539cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cdccc466-x6w9n"
Sep 17 20:55:42.332651 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.332551    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/49d98dab-dc56-41d3-b85b-758096e539cc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76cdccc466-x6w9n\" (UID: \"49d98dab-dc56-41d3-b85b-758096e539cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cdccc466-x6w9n"
Sep 17 20:55:42.332651 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.332578    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/49d98dab-dc56-41d3-b85b-758096e539cc-ca\") pod \"cluster-proxy-proxy-agent-76cdccc466-x6w9n\" (UID: \"49d98dab-dc56-41d3-b85b-758096e539cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cdccc466-x6w9n"
Sep 17 20:55:42.332651 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.332625    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49d98dab-dc56-41d3-b85b-758096e539cc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76cdccc466-x6w9n\" (UID: \"49d98dab-dc56-41d3-b85b-758096e539cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cdccc466-x6w9n"
Sep 17 20:55:42.332804 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.332676    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wstxn\" (UniqueName: \"kubernetes.io/projected/197ddc7b-1ef5-4262-8c1e-2703d131ff5d-kube-api-access-wstxn\") pod \"managed-serviceaccount-addon-agent-7c6c7465c4-2d8kq\" (UID: \"197ddc7b-1ef5-4262-8c1e-2703d131ff5d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c6c7465c4-2d8kq"
Sep 17 20:55:42.332804 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.332749    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/49d98dab-dc56-41d3-b85b-758096e539cc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76cdccc466-x6w9n\" (UID: \"49d98dab-dc56-41d3-b85b-758096e539cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cdccc466-x6w9n"
Sep 17 20:55:42.332804 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.332783    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpjhl\" (UniqueName: \"kubernetes.io/projected/49d98dab-dc56-41d3-b85b-758096e539cc-kube-api-access-qpjhl\") pod \"cluster-proxy-proxy-agent-76cdccc466-x6w9n\" (UID: \"49d98dab-dc56-41d3-b85b-758096e539cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cdccc466-x6w9n"
Sep 17 20:55:42.332948 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.332816    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/197ddc7b-1ef5-4262-8c1e-2703d131ff5d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c6c7465c4-2d8kq\" (UID: \"197ddc7b-1ef5-4262-8c1e-2703d131ff5d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c6c7465c4-2d8kq"
Sep 17 20:55:42.433973 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.433946    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/197ddc7b-1ef5-4262-8c1e-2703d131ff5d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c6c7465c4-2d8kq\" (UID: \"197ddc7b-1ef5-4262-8c1e-2703d131ff5d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c6c7465c4-2d8kq"
Sep 17 20:55:42.434125 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.433978    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/49d98dab-dc56-41d3-b85b-758096e539cc-hub\") pod \"cluster-proxy-proxy-agent-76cdccc466-x6w9n\" (UID: \"49d98dab-dc56-41d3-b85b-758096e539cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cdccc466-x6w9n"
Sep 17 20:55:42.434125 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.433995    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/49d98dab-dc56-41d3-b85b-758096e539cc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76cdccc466-x6w9n\" (UID: \"49d98dab-dc56-41d3-b85b-758096e539cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cdccc466-x6w9n"
Sep 17 20:55:42.434125 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.434118    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/49d98dab-dc56-41d3-b85b-758096e539cc-ca\") pod \"cluster-proxy-proxy-agent-76cdccc466-x6w9n\" (UID: \"49d98dab-dc56-41d3-b85b-758096e539cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cdccc466-x6w9n"
Sep 17 20:55:42.434320 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.434161    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49d98dab-dc56-41d3-b85b-758096e539cc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76cdccc466-x6w9n\" (UID: \"49d98dab-dc56-41d3-b85b-758096e539cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cdccc466-x6w9n"
Sep 17 20:55:42.434320 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.434236    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wstxn\" (UniqueName: \"kubernetes.io/projected/197ddc7b-1ef5-4262-8c1e-2703d131ff5d-kube-api-access-wstxn\") pod \"managed-serviceaccount-addon-agent-7c6c7465c4-2d8kq\" (UID: \"197ddc7b-1ef5-4262-8c1e-2703d131ff5d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c6c7465c4-2d8kq"
Sep 17 20:55:42.434320 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.434293    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/49d98dab-dc56-41d3-b85b-758096e539cc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76cdccc466-x6w9n\" (UID: \"49d98dab-dc56-41d3-b85b-758096e539cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cdccc466-x6w9n"
Sep 17 20:55:42.434320 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.434315    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpjhl\" (UniqueName: \"kubernetes.io/projected/49d98dab-dc56-41d3-b85b-758096e539cc-kube-api-access-qpjhl\") pod \"cluster-proxy-proxy-agent-76cdccc466-x6w9n\" (UID: \"49d98dab-dc56-41d3-b85b-758096e539cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cdccc466-x6w9n"
Sep 17 20:55:42.435077 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.435056    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/49d98dab-dc56-41d3-b85b-758096e539cc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76cdccc466-x6w9n\" (UID: \"49d98dab-dc56-41d3-b85b-758096e539cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cdccc466-x6w9n"
Sep 17 20:55:42.436676 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.436653    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/197ddc7b-1ef5-4262-8c1e-2703d131ff5d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c6c7465c4-2d8kq\" (UID: \"197ddc7b-1ef5-4262-8c1e-2703d131ff5d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c6c7465c4-2d8kq"
Sep 17 20:55:42.436676 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.436667    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/49d98dab-dc56-41d3-b85b-758096e539cc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76cdccc466-x6w9n\" (UID: \"49d98dab-dc56-41d3-b85b-758096e539cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cdccc466-x6w9n"
Sep 17 20:55:42.436826 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.436796    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/49d98dab-dc56-41d3-b85b-758096e539cc-ca\") pod \"cluster-proxy-proxy-agent-76cdccc466-x6w9n\" (UID: \"49d98dab-dc56-41d3-b85b-758096e539cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cdccc466-x6w9n"
Sep 17 20:55:42.436880 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.436871    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/49d98dab-dc56-41d3-b85b-758096e539cc-hub\") pod \"cluster-proxy-proxy-agent-76cdccc466-x6w9n\" (UID: \"49d98dab-dc56-41d3-b85b-758096e539cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cdccc466-x6w9n"
Sep 17 20:55:42.437027 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.437012    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49d98dab-dc56-41d3-b85b-758096e539cc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76cdccc466-x6w9n\" (UID: \"49d98dab-dc56-41d3-b85b-758096e539cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cdccc466-x6w9n"
Sep 17 20:55:42.442001 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.441981    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpjhl\" (UniqueName: \"kubernetes.io/projected/49d98dab-dc56-41d3-b85b-758096e539cc-kube-api-access-qpjhl\") pod \"cluster-proxy-proxy-agent-76cdccc466-x6w9n\" (UID: \"49d98dab-dc56-41d3-b85b-758096e539cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cdccc466-x6w9n"
Sep 17 20:55:42.442308 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.442288    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wstxn\" (UniqueName: \"kubernetes.io/projected/197ddc7b-1ef5-4262-8c1e-2703d131ff5d-kube-api-access-wstxn\") pod \"managed-serviceaccount-addon-agent-7c6c7465c4-2d8kq\" (UID: \"197ddc7b-1ef5-4262-8c1e-2703d131ff5d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c6c7465c4-2d8kq"
Sep 17 20:55:42.569965 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.569893    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c6c7465c4-2d8kq"
Sep 17 20:55:42.622643 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.622620    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cdccc466-x6w9n"
Sep 17 20:55:42.837994 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.837920    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c6c7465c4-2d8kq"]
Sep 17 20:55:42.840913 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:55:42.840891    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod197ddc7b_1ef5_4262_8c1e_2703d131ff5d.slice/crio-c04309abd5896155cf98b186d721a0d9cac9b4de80778670f21b5f3d76f769d1 WatchSource:0}: Error finding container c04309abd5896155cf98b186d721a0d9cac9b4de80778670f21b5f3d76f769d1: Status 404 returned error can't find the container with id c04309abd5896155cf98b186d721a0d9cac9b4de80778670f21b5f3d76f769d1
Sep 17 20:55:42.842740 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.842726    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 20:55:42.871688 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.871662    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c6c7465c4-2d8kq" event={"ID":"197ddc7b-1ef5-4262-8c1e-2703d131ff5d","Type":"ContainerStarted","Data":"c04309abd5896155cf98b186d721a0d9cac9b4de80778670f21b5f3d76f769d1"}
Sep 17 20:55:42.898330 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:42.898302    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cdccc466-x6w9n"]
Sep 17 20:55:42.900163 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:55:42.900142    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49d98dab_dc56_41d3_b85b_758096e539cc.slice/crio-625a61caeba0f21b2ce6d65dda1de046cc5600e83b74aa30be83c9caead4f4fd WatchSource:0}: Error finding container 625a61caeba0f21b2ce6d65dda1de046cc5600e83b74aa30be83c9caead4f4fd: Status 404 returned error can't find the container with id 625a61caeba0f21b2ce6d65dda1de046cc5600e83b74aa30be83c9caead4f4fd
Sep 17 20:55:43.885291 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:43.882398    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cdccc466-x6w9n" event={"ID":"49d98dab-dc56-41d3-b85b-758096e539cc","Type":"ContainerStarted","Data":"625a61caeba0f21b2ce6d65dda1de046cc5600e83b74aa30be83c9caead4f4fd"}
Sep 17 20:55:46.898281 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:46.898244    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cdccc466-x6w9n" event={"ID":"49d98dab-dc56-41d3-b85b-758096e539cc","Type":"ContainerStarted","Data":"430cb7c978b742baf58a80fc4e8a23dc895fec22ef7dc1643f88a892c97052eb"}
Sep 17 20:55:46.898642 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:46.898299    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cdccc466-x6w9n" event={"ID":"49d98dab-dc56-41d3-b85b-758096e539cc","Type":"ContainerStarted","Data":"c036dec8e8ad40bf93750f774a88cde7b6c8c02f789ff5f84ed98abfefff33a1"}
Sep 17 20:55:46.900034 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:46.900006    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c6c7465c4-2d8kq" event={"ID":"197ddc7b-1ef5-4262-8c1e-2703d131ff5d","Type":"ContainerStarted","Data":"4626a74ee5d090afbd0b878c1909cc169f8e2881310bfed11905b30c5e0489e1"}
Sep 17 20:55:46.916480 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:46.916438    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c6c7465c4-2d8kq" podStartSLOduration=1.051639948 podStartE2EDuration="4.916423826s" totalImagesPullingTime="3.864783878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:55:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:55:46.915828425 +0000 UTC m=+388.526561655" watchObservedRunningTime="2026-09-17 20:55:46.916423826 +0000 UTC m=+388.527157054"
Sep 17 20:55:47.906245 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:47.906196    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cdccc466-x6w9n" event={"ID":"49d98dab-dc56-41d3-b85b-758096e539cc","Type":"ContainerStarted","Data":"988986ede7577eebe908bf499b1b2c6a736ab44d277a3175ce7e47dd75778b63"}
Sep 17 20:55:47.926519 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:55:47.926456    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cdccc466-x6w9n" podStartSLOduration=2.062090542 podStartE2EDuration="5.926434816s" totalImagesPullingTime="3.864344274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:55:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:55:47.925513309 +0000 UTC m=+389.536246537" watchObservedRunningTime="2026-09-17 20:55:47.926434816 +0000 UTC m=+389.537168043"
Sep 17 20:57:08.056389 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:08.056357    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-694ff8fcb5-jscsd"]
Sep 17 20:57:33.076513 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:33.076457    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-694ff8fcb5-jscsd" podUID="c347dc67-05b9-4b92-9609-bc8a09b9ad7e" containerName="console" containerID="cri-o://b128501103b7dfe3eb35fde64dfd2f975cca22cb662b386deadf131e62e842d7" gracePeriod=15
Sep 17 20:57:33.290772 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:33.290741    2566 generic.go:356] "Generic (PLEG): container finished" podID="c347dc67-05b9-4b92-9609-bc8a09b9ad7e" containerID="b128501103b7dfe3eb35fde64dfd2f975cca22cb662b386deadf131e62e842d7" exitCode=0
Sep 17 20:57:33.290909 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:33.290815    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-694ff8fcb5-jscsd" event={"ID":"c347dc67-05b9-4b92-9609-bc8a09b9ad7e","Type":"ContainerDied","Data":"b128501103b7dfe3eb35fde64dfd2f975cca22cb662b386deadf131e62e842d7"}
Sep 17 20:57:33.309191 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:33.309171    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-694ff8fcb5-jscsd"
Sep 17 20:57:33.444976 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:33.444889    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-trusted-ca-bundle\") pod \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\" (UID: \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\") "
Sep 17 20:57:33.444976 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:33.444926    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-console-config\") pod \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\" (UID: \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\") "
Sep 17 20:57:33.444976 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:33.444950    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-console-oauth-config\") pod \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\" (UID: \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\") "
Sep 17 20:57:33.444976 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:33.444973    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-console-serving-cert\") pod \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\" (UID: \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\") "
Sep 17 20:57:33.445320 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:33.445014    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkxpc\" (UniqueName: \"kubernetes.io/projected/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-kube-api-access-tkxpc\") pod \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\" (UID: \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\") "
Sep 17 20:57:33.445320 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:33.445043    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-service-ca\") pod \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\" (UID: \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\") "
Sep 17 20:57:33.445320 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:33.445068    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-oauth-serving-cert\") pod \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\" (UID: \"c347dc67-05b9-4b92-9609-bc8a09b9ad7e\") "
Sep 17 20:57:33.445470 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:33.445398    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-console-config" (OuterVolumeSpecName: "console-config") pod "c347dc67-05b9-4b92-9609-bc8a09b9ad7e" (UID: "c347dc67-05b9-4b92-9609-bc8a09b9ad7e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:57:33.445523 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:33.445499    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-service-ca" (OuterVolumeSpecName: "service-ca") pod "c347dc67-05b9-4b92-9609-bc8a09b9ad7e" (UID: "c347dc67-05b9-4b92-9609-bc8a09b9ad7e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:57:33.445578 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:33.445549    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c347dc67-05b9-4b92-9609-bc8a09b9ad7e" (UID: "c347dc67-05b9-4b92-9609-bc8a09b9ad7e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:57:33.445631 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:33.445587    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c347dc67-05b9-4b92-9609-bc8a09b9ad7e" (UID: "c347dc67-05b9-4b92-9609-bc8a09b9ad7e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:57:33.447407 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:33.447375    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c347dc67-05b9-4b92-9609-bc8a09b9ad7e" (UID: "c347dc67-05b9-4b92-9609-bc8a09b9ad7e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:57:33.447717 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:33.447670    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-kube-api-access-tkxpc" (OuterVolumeSpecName: "kube-api-access-tkxpc") pod "c347dc67-05b9-4b92-9609-bc8a09b9ad7e" (UID: "c347dc67-05b9-4b92-9609-bc8a09b9ad7e"). InnerVolumeSpecName "kube-api-access-tkxpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:57:33.447717 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:33.447679    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c347dc67-05b9-4b92-9609-bc8a09b9ad7e" (UID: "c347dc67-05b9-4b92-9609-bc8a09b9ad7e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:57:33.546464 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:33.546434    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkxpc\" (UniqueName: \"kubernetes.io/projected/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-kube-api-access-tkxpc\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:57:33.546464 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:33.546464    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-service-ca\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:57:33.546642 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:33.546474    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-oauth-serving-cert\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:57:33.546642 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:33.546502    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-trusted-ca-bundle\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:57:33.546642 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:33.546511    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-console-config\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:57:33.546642 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:33.546538    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-console-oauth-config\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:57:33.546642 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:33.546547    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c347dc67-05b9-4b92-9609-bc8a09b9ad7e-console-serving-cert\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 20:57:34.295572 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:34.295542    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-694ff8fcb5-jscsd"
Sep 17 20:57:34.295976 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:34.295550    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-694ff8fcb5-jscsd" event={"ID":"c347dc67-05b9-4b92-9609-bc8a09b9ad7e","Type":"ContainerDied","Data":"f04ee81d0a2289d5b9a1dd29644fdc4ce0dc1e9f6e64d50fa7aa6fe7e921fb5f"}
Sep 17 20:57:34.295976 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:34.295660    2566 scope.go:117] "RemoveContainer" containerID="b128501103b7dfe3eb35fde64dfd2f975cca22cb662b386deadf131e62e842d7"
Sep 17 20:57:34.350133 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:34.350103    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-694ff8fcb5-jscsd"]
Sep 17 20:57:34.382431 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:34.382407    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-694ff8fcb5-jscsd"]
Sep 17 20:57:34.462245 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:57:34.462199    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c347dc67-05b9-4b92-9609-bc8a09b9ad7e" path="/var/lib/kubelet/pods/c347dc67-05b9-4b92-9609-bc8a09b9ad7e/volumes"
Sep 17 20:59:20.391095 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:59:20.391066    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t2rlv_ed6472a6-7979-43b0-a549-7ca5c5d8fb0f/console-operator/1.log"
Sep 17 20:59:20.391742 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:59:20.391723    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t2rlv_ed6472a6-7979-43b0-a549-7ca5c5d8fb0f/console-operator/1.log"
Sep 17 20:59:20.396246 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:59:20.396227    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnqx7_402f55a4-24b6-47c6-b719-9c6a5b473434/ovn-acl-logging/0.log"
Sep 17 20:59:20.396923 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:59:20.396906    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnqx7_402f55a4-24b6-47c6-b719-9c6a5b473434/ovn-acl-logging/0.log"
Sep 17 20:59:35.351718 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:59:35.351642    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-gcv9m"]
Sep 17 20:59:35.352189 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:59:35.351927    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c347dc67-05b9-4b92-9609-bc8a09b9ad7e" containerName="console"
Sep 17 20:59:35.352189 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:59:35.351939    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c347dc67-05b9-4b92-9609-bc8a09b9ad7e" containerName="console"
Sep 17 20:59:35.352189 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:59:35.351997    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c347dc67-05b9-4b92-9609-bc8a09b9ad7e" containerName="console"
Sep 17 20:59:35.354816 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:59:35.354802    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-gcv9m"
Sep 17 20:59:35.357534 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:59:35.357514    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 20:59:35.357534 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:59:35.357522    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-tm42p\""
Sep 17 20:59:35.357705 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:59:35.357523    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 20:59:35.358685 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:59:35.358667    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 20:59:35.358866 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:59:35.358854    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 20:59:35.363244 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:59:35.363202    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-gcv9m"]
Sep 17 20:59:35.524895 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:59:35.524863    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9-model-cache\") pod \"s3-init-gcv9m\" (UID: \"0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9\") " pod="kserve/s3-init-gcv9m"
Sep 17 20:59:35.524895 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:59:35.524897    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-487j5\" (UniqueName: \"kubernetes.io/projected/0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9-kube-api-access-487j5\") pod \"s3-init-gcv9m\" (UID: \"0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9\") " pod="kserve/s3-init-gcv9m"
Sep 17 20:59:35.625834 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:59:35.625753    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9-model-cache\") pod \"s3-init-gcv9m\" (UID: \"0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9\") " pod="kserve/s3-init-gcv9m"
Sep 17 20:59:35.625834 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:59:35.625789    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-487j5\" (UniqueName: \"kubernetes.io/projected/0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9-kube-api-access-487j5\") pod \"s3-init-gcv9m\" (UID: \"0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9\") " pod="kserve/s3-init-gcv9m"
Sep 17 20:59:35.626329 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:59:35.626308    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9-model-cache\") pod \"s3-init-gcv9m\" (UID: \"0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9\") " pod="kserve/s3-init-gcv9m"
Sep 17 20:59:35.634196 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:59:35.634173    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-487j5\" (UniqueName: \"kubernetes.io/projected/0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9-kube-api-access-487j5\") pod \"s3-init-gcv9m\" (UID: \"0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9\") " pod="kserve/s3-init-gcv9m"
Sep 17 20:59:35.664153 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:59:35.664128    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-gcv9m"
Sep 17 20:59:35.931660 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:59:35.931595    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-gcv9m"]
Sep 17 20:59:35.934139 ip-10-0-138-140 kubenswrapper[2566]: W0917 20:59:35.934110    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ff37ebc_9a3c_4c6d_8d33_a6b6bb7863d9.slice/crio-e8fbac5dcd9ec6f7b3f60bbb802d5507345fd0ace28a44be7aa4612b4899a7a4 WatchSource:0}: Error finding container e8fbac5dcd9ec6f7b3f60bbb802d5507345fd0ace28a44be7aa4612b4899a7a4: Status 404 returned error can't find the container with id e8fbac5dcd9ec6f7b3f60bbb802d5507345fd0ace28a44be7aa4612b4899a7a4
Sep 17 20:59:36.746269 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:59:36.746233    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gcv9m" event={"ID":"0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9","Type":"ContainerStarted","Data":"e8fbac5dcd9ec6f7b3f60bbb802d5507345fd0ace28a44be7aa4612b4899a7a4"}
Sep 17 20:59:39.764864 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:59:39.764823    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gcv9m" event={"ID":"0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9","Type":"ContainerStarted","Data":"a0f21929643c8ae193378bb42a5e30d47525e5271f7805f6aadbb1a077e14279"}
Sep 17 20:59:41.776725 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:59:41.776696    2566 generic.go:356] "Generic (PLEG): container finished" podID="0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9" containerID="a0f21929643c8ae193378bb42a5e30d47525e5271f7805f6aadbb1a077e14279" exitCode=0
Sep 17 20:59:41.777092 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:59:41.776740    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gcv9m" event={"ID":"0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9","Type":"ContainerDied","Data":"a0f21929643c8ae193378bb42a5e30d47525e5271f7805f6aadbb1a077e14279"}
Sep 17 20:59:46.802427 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:59:46.802371    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gcv9m" event={"ID":"0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9","Type":"ContainerStarted","Data":"21e78b56a49b0540d2e01ca573c10b820b8e25e4039c610bfeb185340259c5b8"}
Sep 17 20:59:46.822549 ip-10-0-138-140 kubenswrapper[2566]: I0917 20:59:46.822490    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-gcv9m" podStartSLOduration=2.177574443 podStartE2EDuration="11.822470788s" totalImagesPullingTime="7.872672383s" totalInitContainerRuntime="1.772223962s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:59:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:59:46.820819972 +0000 UTC m=+628.431553201" watchObservedRunningTime="2026-09-17 20:59:46.822470788 +0000 UTC m=+628.433204021"
Sep 17 21:00:52.054144 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:00:52.054107    2566 generic.go:356] "Generic (PLEG): container finished" podID="0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9" containerID="21e78b56a49b0540d2e01ca573c10b820b8e25e4039c610bfeb185340259c5b8" exitCode=0
Sep 17 21:00:52.054595 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:00:52.054178    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gcv9m" event={"ID":"0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9","Type":"ContainerDied","Data":"21e78b56a49b0540d2e01ca573c10b820b8e25e4039c610bfeb185340259c5b8"}
Sep 17 21:00:53.183462 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:00:53.183436    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-gcv9m"
Sep 17 21:00:53.225099 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:00:53.225077    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-487j5\" (UniqueName: \"kubernetes.io/projected/0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9-kube-api-access-487j5\") pod \"0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9\" (UID: \"0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9\") "
Sep 17 21:00:53.225240 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:00:53.225151    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9-model-cache\") pod \"0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9\" (UID: \"0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9\") "
Sep 17 21:00:53.227177 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:00:53.227146    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9-kube-api-access-487j5" (OuterVolumeSpecName: "kube-api-access-487j5") pod "0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9" (UID: "0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9"). InnerVolumeSpecName "kube-api-access-487j5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:00:53.244402 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:00:53.244204    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9" (UID: "0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:00:53.325790 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:00:53.325739    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9-model-cache\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:00:53.325790 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:00:53.325759    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-487j5\" (UniqueName: \"kubernetes.io/projected/0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9-kube-api-access-487j5\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:00:54.062704 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:00:54.062676    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-gcv9m"
Sep 17 21:00:54.062871 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:00:54.062704    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gcv9m" event={"ID":"0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9","Type":"ContainerDied","Data":"e8fbac5dcd9ec6f7b3f60bbb802d5507345fd0ace28a44be7aa4612b4899a7a4"}
Sep 17 21:00:54.062871 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:00:54.062735    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8fbac5dcd9ec6f7b3f60bbb802d5507345fd0ace28a44be7aa4612b4899a7a4"
Sep 17 21:01:07.369762 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.369690    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2"]
Sep 17 21:01:07.370152 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.370023    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9" containerName="s3-init"
Sep 17 21:01:07.370152 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.370039    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9" containerName="s3-init"
Sep 17 21:01:07.370152 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.370051    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9" containerName="download-hf-model"
Sep 17 21:01:07.370152 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.370057    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9" containerName="download-hf-model"
Sep 17 21:01:07.370152 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.370126    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9" containerName="s3-init"
Sep 17 21:01:07.373176 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.373161    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2"
Sep 17 21:01:07.375915 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.375896    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 17 21:01:07.376129 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.376112    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 21:01:07.376252 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.376114    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:01:07.377006 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.376976    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 21:01:07.377006 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.376993    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5dj6n\""
Sep 17 21:01:07.377132 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.377005    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 21:01:07.377132 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.377086    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 21:01:07.389691 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.389671    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2"]
Sep 17 21:01:07.429811 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.429785    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7d49a27-d465-49af-8f11-11d586d53a21-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2\" (UID: \"e7d49a27-d465-49af-8f11-11d586d53a21\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2"
Sep 17 21:01:07.429939 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.429835    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7d49a27-d465-49af-8f11-11d586d53a21-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2\" (UID: \"e7d49a27-d465-49af-8f11-11d586d53a21\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2"
Sep 17 21:01:07.429939 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.429885    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9sjl\" (UniqueName: \"kubernetes.io/projected/e7d49a27-d465-49af-8f11-11d586d53a21-kube-api-access-x9sjl\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2\" (UID: \"e7d49a27-d465-49af-8f11-11d586d53a21\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2"
Sep 17 21:01:07.430017 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.429986    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7d49a27-d465-49af-8f11-11d586d53a21-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2\" (UID: \"e7d49a27-d465-49af-8f11-11d586d53a21\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2"
Sep 17 21:01:07.430059 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.430028    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7d49a27-d465-49af-8f11-11d586d53a21-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2\" (UID: \"e7d49a27-d465-49af-8f11-11d586d53a21\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2"
Sep 17 21:01:07.452115 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.452086    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq"]
Sep 17 21:01:07.455441 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.455427    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq"
Sep 17 21:01:07.457964 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.457943    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 21:01:07.482701 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.482675    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq"]
Sep 17 21:01:07.531063 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.531036    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f2467cf-97ac-40f4-8c8c-3468b677908c-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-4dlsq\" (UID: \"3f2467cf-97ac-40f4-8c8c-3468b677908c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq"
Sep 17 21:01:07.531262 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.531072    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f2467cf-97ac-40f4-8c8c-3468b677908c-cabundle-cert\") pod \"isvc-canary-rollback-predictor-97977b6b5-4dlsq\" (UID: \"3f2467cf-97ac-40f4-8c8c-3468b677908c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq"
Sep 17 21:01:07.531262 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.531093    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7d49a27-d465-49af-8f11-11d586d53a21-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2\" (UID: \"e7d49a27-d465-49af-8f11-11d586d53a21\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2"
Sep 17 21:01:07.531262 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.531158    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f2467cf-97ac-40f4-8c8c-3468b677908c-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-97977b6b5-4dlsq\" (UID: \"3f2467cf-97ac-40f4-8c8c-3468b677908c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq"
Sep 17 21:01:07.531262 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.531190    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9sjl\" (UniqueName: \"kubernetes.io/projected/e7d49a27-d465-49af-8f11-11d586d53a21-kube-api-access-x9sjl\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2\" (UID: \"e7d49a27-d465-49af-8f11-11d586d53a21\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2"
Sep 17 21:01:07.531262 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.531230    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7d49a27-d465-49af-8f11-11d586d53a21-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2\" (UID: \"e7d49a27-d465-49af-8f11-11d586d53a21\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2"
Sep 17 21:01:07.531262 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.531252    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7d49a27-d465-49af-8f11-11d586d53a21-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2\" (UID: \"e7d49a27-d465-49af-8f11-11d586d53a21\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2"
Sep 17 21:01:07.531565 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.531286    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfk9x\" (UniqueName: \"kubernetes.io/projected/3f2467cf-97ac-40f4-8c8c-3468b677908c-kube-api-access-pfk9x\") pod \"isvc-canary-rollback-predictor-97977b6b5-4dlsq\" (UID: \"3f2467cf-97ac-40f4-8c8c-3468b677908c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq"
Sep 17 21:01:07.531565 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.531318    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7d49a27-d465-49af-8f11-11d586d53a21-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2\" (UID: \"e7d49a27-d465-49af-8f11-11d586d53a21\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2"
Sep 17 21:01:07.531565 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.531353    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f2467cf-97ac-40f4-8c8c-3468b677908c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-97977b6b5-4dlsq\" (UID: \"3f2467cf-97ac-40f4-8c8c-3468b677908c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq"
Sep 17 21:01:07.531722 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.531602    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7d49a27-d465-49af-8f11-11d586d53a21-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2\" (UID: \"e7d49a27-d465-49af-8f11-11d586d53a21\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2"
Sep 17 21:01:07.531775 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.531762    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7d49a27-d465-49af-8f11-11d586d53a21-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2\" (UID: \"e7d49a27-d465-49af-8f11-11d586d53a21\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2"
Sep 17 21:01:07.531843 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.531824    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7d49a27-d465-49af-8f11-11d586d53a21-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2\" (UID: \"e7d49a27-d465-49af-8f11-11d586d53a21\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2"
Sep 17 21:01:07.533647 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.533627    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7d49a27-d465-49af-8f11-11d586d53a21-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2\" (UID: \"e7d49a27-d465-49af-8f11-11d586d53a21\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2"
Sep 17 21:01:07.543948 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.543923    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9sjl\" (UniqueName: \"kubernetes.io/projected/e7d49a27-d465-49af-8f11-11d586d53a21-kube-api-access-x9sjl\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2\" (UID: \"e7d49a27-d465-49af-8f11-11d586d53a21\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2"
Sep 17 21:01:07.572247 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.572224    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"]
Sep 17 21:01:07.575861 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.575846    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"
Sep 17 21:01:07.579302 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.579219    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-89dfa-kube-rbac-proxy-sar-config\""
Sep 17 21:01:07.579742 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.579727    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-89dfa-predictor-serving-cert\""
Sep 17 21:01:07.592584 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.592564    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"]
Sep 17 21:01:07.632116 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.632058    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f2467cf-97ac-40f4-8c8c-3468b677908c-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-97977b6b5-4dlsq\" (UID: \"3f2467cf-97ac-40f4-8c8c-3468b677908c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq"
Sep 17 21:01:07.632116 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.632092    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jjmf\" (UniqueName: \"kubernetes.io/projected/6a308e40-a9f1-4a70-aae9-eca1b7b30d37-kube-api-access-9jjmf\") pod \"isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl\" (UID: \"6a308e40-a9f1-4a70-aae9-eca1b7b30d37\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"
Sep 17 21:01:07.632116 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.632115    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a308e40-a9f1-4a70-aae9-eca1b7b30d37-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl\" (UID: \"6a308e40-a9f1-4a70-aae9-eca1b7b30d37\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"
Sep 17 21:01:07.632308 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.632143    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfk9x\" (UniqueName: \"kubernetes.io/projected/3f2467cf-97ac-40f4-8c8c-3468b677908c-kube-api-access-pfk9x\") pod \"isvc-canary-rollback-predictor-97977b6b5-4dlsq\" (UID: \"3f2467cf-97ac-40f4-8c8c-3468b677908c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq"
Sep 17 21:01:07.632308 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.632186    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a308e40-a9f1-4a70-aae9-eca1b7b30d37-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl\" (UID: \"6a308e40-a9f1-4a70-aae9-eca1b7b30d37\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"
Sep 17 21:01:07.632308 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.632242    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f2467cf-97ac-40f4-8c8c-3468b677908c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-97977b6b5-4dlsq\" (UID: \"3f2467cf-97ac-40f4-8c8c-3468b677908c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq"
Sep 17 21:01:07.632308 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.632278    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f2467cf-97ac-40f4-8c8c-3468b677908c-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-4dlsq\" (UID: \"3f2467cf-97ac-40f4-8c8c-3468b677908c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq"
Sep 17 21:01:07.632476 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.632343    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f2467cf-97ac-40f4-8c8c-3468b677908c-cabundle-cert\") pod \"isvc-canary-rollback-predictor-97977b6b5-4dlsq\" (UID: \"3f2467cf-97ac-40f4-8c8c-3468b677908c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq"
Sep 17 21:01:07.632476 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.632377    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a308e40-a9f1-4a70-aae9-eca1b7b30d37-proxy-tls\") pod \"isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl\" (UID: \"6a308e40-a9f1-4a70-aae9-eca1b7b30d37\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"
Sep 17 21:01:07.632476 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.632412    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a308e40-a9f1-4a70-aae9-eca1b7b30d37-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl\" (UID: \"6a308e40-a9f1-4a70-aae9-eca1b7b30d37\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"
Sep 17 21:01:07.632476 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.632434    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f2467cf-97ac-40f4-8c8c-3468b677908c-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-97977b6b5-4dlsq\" (UID: \"3f2467cf-97ac-40f4-8c8c-3468b677908c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq"
Sep 17 21:01:07.632883 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.632856    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f2467cf-97ac-40f4-8c8c-3468b677908c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-97977b6b5-4dlsq\" (UID: \"3f2467cf-97ac-40f4-8c8c-3468b677908c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq"
Sep 17 21:01:07.632999 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.632984    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f2467cf-97ac-40f4-8c8c-3468b677908c-cabundle-cert\") pod \"isvc-canary-rollback-predictor-97977b6b5-4dlsq\" (UID: \"3f2467cf-97ac-40f4-8c8c-3468b677908c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq"
Sep 17 21:01:07.634711 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.634693    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f2467cf-97ac-40f4-8c8c-3468b677908c-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-4dlsq\" (UID: \"3f2467cf-97ac-40f4-8c8c-3468b677908c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq"
Sep 17 21:01:07.645179 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.645162    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfk9x\" (UniqueName: \"kubernetes.io/projected/3f2467cf-97ac-40f4-8c8c-3468b677908c-kube-api-access-pfk9x\") pod \"isvc-canary-rollback-predictor-97977b6b5-4dlsq\" (UID: \"3f2467cf-97ac-40f4-8c8c-3468b677908c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq"
Sep 17 21:01:07.683074 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.683054    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2"
Sep 17 21:01:07.733825 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.733776    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a308e40-a9f1-4a70-aae9-eca1b7b30d37-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl\" (UID: \"6a308e40-a9f1-4a70-aae9-eca1b7b30d37\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"
Sep 17 21:01:07.733950 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.733872    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a308e40-a9f1-4a70-aae9-eca1b7b30d37-proxy-tls\") pod \"isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl\" (UID: \"6a308e40-a9f1-4a70-aae9-eca1b7b30d37\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"
Sep 17 21:01:07.733950 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.733907    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a308e40-a9f1-4a70-aae9-eca1b7b30d37-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl\" (UID: \"6a308e40-a9f1-4a70-aae9-eca1b7b30d37\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"
Sep 17 21:01:07.734061 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.733953    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jjmf\" (UniqueName: \"kubernetes.io/projected/6a308e40-a9f1-4a70-aae9-eca1b7b30d37-kube-api-access-9jjmf\") pod \"isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl\" (UID: \"6a308e40-a9f1-4a70-aae9-eca1b7b30d37\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"
Sep 17 21:01:07.734061 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.733991    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a308e40-a9f1-4a70-aae9-eca1b7b30d37-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl\" (UID: \"6a308e40-a9f1-4a70-aae9-eca1b7b30d37\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"
Sep 17 21:01:07.734597 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.734572    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a308e40-a9f1-4a70-aae9-eca1b7b30d37-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl\" (UID: \"6a308e40-a9f1-4a70-aae9-eca1b7b30d37\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"
Sep 17 21:01:07.734740 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.734701    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a308e40-a9f1-4a70-aae9-eca1b7b30d37-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl\" (UID: \"6a308e40-a9f1-4a70-aae9-eca1b7b30d37\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"
Sep 17 21:01:07.735019 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.734977    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a308e40-a9f1-4a70-aae9-eca1b7b30d37-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl\" (UID: \"6a308e40-a9f1-4a70-aae9-eca1b7b30d37\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"
Sep 17 21:01:07.737322 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.737299    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a308e40-a9f1-4a70-aae9-eca1b7b30d37-proxy-tls\") pod \"isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl\" (UID: \"6a308e40-a9f1-4a70-aae9-eca1b7b30d37\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"
Sep 17 21:01:07.745591 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.745557    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jjmf\" (UniqueName: \"kubernetes.io/projected/6a308e40-a9f1-4a70-aae9-eca1b7b30d37-kube-api-access-9jjmf\") pod \"isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl\" (UID: \"6a308e40-a9f1-4a70-aae9-eca1b7b30d37\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"
Sep 17 21:01:07.765340 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.765319    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq"
Sep 17 21:01:07.886474 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.886388    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"
Sep 17 21:01:07.954935 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:07.954912    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2"]
Sep 17 21:01:07.957081 ip-10-0-138-140 kubenswrapper[2566]: W0917 21:01:07.957052    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d49a27_d465_49af_8f11_11d586d53a21.slice/crio-73868b5686fe0d01fb7312abd37285c652fdcc8c38ef0971bd943f42dc416e8a WatchSource:0}: Error finding container 73868b5686fe0d01fb7312abd37285c652fdcc8c38ef0971bd943f42dc416e8a: Status 404 returned error can't find the container with id 73868b5686fe0d01fb7312abd37285c652fdcc8c38ef0971bd943f42dc416e8a
Sep 17 21:01:08.036917 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:08.036889    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq"]
Sep 17 21:01:08.038797 ip-10-0-138-140 kubenswrapper[2566]: W0917 21:01:08.038772    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f2467cf_97ac_40f4_8c8c_3468b677908c.slice/crio-5a4970d55d3de54a23b1cf8a9aa094f4c88518e98a1e7f77eb6ccf3f38bce6df WatchSource:0}: Error finding container 5a4970d55d3de54a23b1cf8a9aa094f4c88518e98a1e7f77eb6ccf3f38bce6df: Status 404 returned error can't find the container with id 5a4970d55d3de54a23b1cf8a9aa094f4c88518e98a1e7f77eb6ccf3f38bce6df
Sep 17 21:01:08.118636 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:08.118598    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2" event={"ID":"e7d49a27-d465-49af-8f11-11d586d53a21","Type":"ContainerStarted","Data":"a0583feeab1e4e1cc75c148eaaf698e3394aa211f02c93774371ae8e2655ae4f"}
Sep 17 21:01:08.118782 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:08.118724    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2" event={"ID":"e7d49a27-d465-49af-8f11-11d586d53a21","Type":"ContainerStarted","Data":"73868b5686fe0d01fb7312abd37285c652fdcc8c38ef0971bd943f42dc416e8a"}
Sep 17 21:01:08.120552 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:08.120528    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq" event={"ID":"3f2467cf-97ac-40f4-8c8c-3468b677908c","Type":"ContainerStarted","Data":"6ba3d87ff194470158ead06f862c8cc7effb844bfe690747a4c898af59cd3fc0"}
Sep 17 21:01:08.120675 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:08.120606    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq" event={"ID":"3f2467cf-97ac-40f4-8c8c-3468b677908c","Type":"ContainerStarted","Data":"5a4970d55d3de54a23b1cf8a9aa094f4c88518e98a1e7f77eb6ccf3f38bce6df"}
Sep 17 21:01:08.169970 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:08.169933    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"]
Sep 17 21:01:08.173493 ip-10-0-138-140 kubenswrapper[2566]: W0917 21:01:08.173470    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a308e40_a9f1_4a70_aae9_eca1b7b30d37.slice/crio-110fdb66c4b7eeb167c47c6f644b9372eb8feb2f251931a3c8c4025b7ce96330 WatchSource:0}: Error finding container 110fdb66c4b7eeb167c47c6f644b9372eb8feb2f251931a3c8c4025b7ce96330: Status 404 returned error can't find the container with id 110fdb66c4b7eeb167c47c6f644b9372eb8feb2f251931a3c8c4025b7ce96330
Sep 17 21:01:09.126070 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:09.126035    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" event={"ID":"6a308e40-a9f1-4a70-aae9-eca1b7b30d37","Type":"ContainerStarted","Data":"8a79d8175f9bc2701536d22ca4191ac3466a7c7b006075d8cbfbf9f04f301c45"}
Sep 17 21:01:09.126574 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:09.126109    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" event={"ID":"6a308e40-a9f1-4a70-aae9-eca1b7b30d37","Type":"ContainerStarted","Data":"110fdb66c4b7eeb167c47c6f644b9372eb8feb2f251931a3c8c4025b7ce96330"}
Sep 17 21:01:12.138326 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:12.138231    2566 generic.go:356] "Generic (PLEG): container finished" podID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerID="8a79d8175f9bc2701536d22ca4191ac3466a7c7b006075d8cbfbf9f04f301c45" exitCode=0
Sep 17 21:01:12.138326 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:12.138245    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" event={"ID":"6a308e40-a9f1-4a70-aae9-eca1b7b30d37","Type":"ContainerDied","Data":"8a79d8175f9bc2701536d22ca4191ac3466a7c7b006075d8cbfbf9f04f301c45"}
Sep 17 21:01:12.139583 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:12.139564    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:01:12.139964 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:12.139938    2566 generic.go:356] "Generic (PLEG): container finished" podID="e7d49a27-d465-49af-8f11-11d586d53a21" containerID="a0583feeab1e4e1cc75c148eaaf698e3394aa211f02c93774371ae8e2655ae4f" exitCode=0
Sep 17 21:01:12.140057 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:12.140017    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2" event={"ID":"e7d49a27-d465-49af-8f11-11d586d53a21","Type":"ContainerDied","Data":"a0583feeab1e4e1cc75c148eaaf698e3394aa211f02c93774371ae8e2655ae4f"}
Sep 17 21:01:12.141620 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:12.141603    2566 generic.go:356] "Generic (PLEG): container finished" podID="3f2467cf-97ac-40f4-8c8c-3468b677908c" containerID="6ba3d87ff194470158ead06f862c8cc7effb844bfe690747a4c898af59cd3fc0" exitCode=0
Sep 17 21:01:12.141669 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:12.141637    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq" event={"ID":"3f2467cf-97ac-40f4-8c8c-3468b677908c","Type":"ContainerDied","Data":"6ba3d87ff194470158ead06f862c8cc7effb844bfe690747a4c898af59cd3fc0"}
Sep 17 21:01:31.308410 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:31.308378    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq" event={"ID":"3f2467cf-97ac-40f4-8c8c-3468b677908c","Type":"ContainerStarted","Data":"fdd6e92f65fe23ea7b109918511e8b6c227aab244a496b51503ace665d737ec2"}
Sep 17 21:01:31.310268 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:31.310245    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" event={"ID":"6a308e40-a9f1-4a70-aae9-eca1b7b30d37","Type":"ContainerStarted","Data":"38374de9970d8416223fae0074418b135a2f895fce2f1f0d7c7c678f2f5b457d"}
Sep 17 21:01:31.312044 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:31.312017    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2" event={"ID":"e7d49a27-d465-49af-8f11-11d586d53a21","Type":"ContainerStarted","Data":"c0bd42eca8bc008c533578e3fe50e8430e80a4efbd9339975cd3d18ef4e29c1d"}
Sep 17 21:01:34.328226 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:34.328186    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" event={"ID":"6a308e40-a9f1-4a70-aae9-eca1b7b30d37","Type":"ContainerStarted","Data":"291a59fd04f0a1ff0f1e49d639e48351af2b1b2f91fd90a109f99fe4468c5b1a"}
Sep 17 21:01:34.330231 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:34.330191    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2" event={"ID":"e7d49a27-d465-49af-8f11-11d586d53a21","Type":"ContainerStarted","Data":"f6d954088c41858a672bd13298c4398a904e8670f4be437ab4e124cae51c3de9"}
Sep 17 21:01:34.330408 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:34.330388    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2"
Sep 17 21:01:34.330525 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:34.330452    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2"
Sep 17 21:01:34.331725 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:34.331689    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2" podUID="e7d49a27-d465-49af-8f11-11d586d53a21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 21:01:34.332440 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:34.332423    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq" event={"ID":"3f2467cf-97ac-40f4-8c8c-3468b677908c","Type":"ContainerStarted","Data":"8f85d941c196626e2e929ad9109f4b4798afcd56b5126ed58306563ae610fecf"}
Sep 17 21:01:34.332581 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:34.332564    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq"
Sep 17 21:01:34.381621 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:34.381540    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq" podStartSLOduration=1.612991388 podStartE2EDuration="27.381526635s" totalImagesPullingTime="21.917849138s" totalInitContainerRuntime="3.850686109s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:01:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:01:34.380467899 +0000 UTC m=+735.991201120" watchObservedRunningTime="2026-09-17 21:01:34.381526635 +0000 UTC m=+735.992259865"
Sep 17 21:01:34.382715 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:34.382678    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2" podStartSLOduration=1.6295163270000002 podStartE2EDuration="27.382667689s" totalImagesPullingTime="21.97847292s" totalInitContainerRuntime="3.774678442s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:01:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:01:34.358478111 +0000 UTC m=+735.969211341" watchObservedRunningTime="2026-09-17 21:01:34.382667689 +0000 UTC m=+735.993400919"
Sep 17 21:01:35.339277 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:35.339074    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2" podUID="e7d49a27-d465-49af-8f11-11d586d53a21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 21:01:35.339277 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:35.339123    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq"
Sep 17 21:01:35.340086 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:35.340056    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq" podUID="3f2467cf-97ac-40f4-8c8c-3468b677908c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 21:01:36.343496 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:36.343457    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq" podUID="3f2467cf-97ac-40f4-8c8c-3468b677908c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 21:01:37.351043 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:37.351009    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" event={"ID":"6a308e40-a9f1-4a70-aae9-eca1b7b30d37","Type":"ContainerStarted","Data":"404401654b48b62c7b9045f9df573666c68d292204732f05ca6af63f995e3cce"}
Sep 17 21:01:37.351428 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:37.351160    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"
Sep 17 21:01:37.351428 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:37.351225    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"
Sep 17 21:01:37.352489 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:37.352463    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 21:01:37.380458 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:37.380411    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podStartSLOduration=2.090030547 podStartE2EDuration="30.380398244s" totalImagesPullingTime="24.6252005s" totalInitContainerRuntime="3.665167197s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:01:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:01:37.378806165 +0000 UTC m=+738.989539395" watchObservedRunningTime="2026-09-17 21:01:37.380398244 +0000 UTC m=+738.991131524"
Sep 17 21:01:38.354758 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:38.354722    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"
Sep 17 21:01:38.355192 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:38.354936    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 21:01:38.355948 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:38.355918    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:01:39.358508 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:39.358434    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 21:01:39.358952 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:39.358845    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:01:39.362478 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:39.362441    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"
Sep 17 21:01:40.343604 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:40.343574    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2"
Sep 17 21:01:40.344003 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:40.343973    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2" podUID="e7d49a27-d465-49af-8f11-11d586d53a21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 21:01:40.362607 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:40.362580    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 21:01:40.362951 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:40.362871    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:01:41.347753 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:41.347726    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq"
Sep 17 21:01:41.348253 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:41.348231    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq" podUID="3f2467cf-97ac-40f4-8c8c-3468b677908c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 21:01:50.344629 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:50.344591    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2" podUID="e7d49a27-d465-49af-8f11-11d586d53a21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 21:01:50.363509 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:50.363478    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 21:01:50.363966 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:50.363947    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:01:51.348567 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:01:51.348533    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq" podUID="3f2467cf-97ac-40f4-8c8c-3468b677908c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 21:02:00.344424 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:02:00.344383    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2" podUID="e7d49a27-d465-49af-8f11-11d586d53a21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 21:02:00.363427 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:02:00.363393    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 21:02:00.363932 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:02:00.363912    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:02:01.348824 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:02:01.348783    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq" podUID="3f2467cf-97ac-40f4-8c8c-3468b677908c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 21:02:10.344337 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:02:10.344297    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2" podUID="e7d49a27-d465-49af-8f11-11d586d53a21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 21:02:10.362797 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:02:10.362767    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 21:02:10.363251 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:02:10.363227    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:02:11.348311 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:02:11.348276    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq" podUID="3f2467cf-97ac-40f4-8c8c-3468b677908c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 21:02:20.344743 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:02:20.344701    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2" podUID="e7d49a27-d465-49af-8f11-11d586d53a21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 21:02:20.363428 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:02:20.363398    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 21:02:20.363817 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:02:20.363790    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:02:21.348387 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:02:21.348351    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq" podUID="3f2467cf-97ac-40f4-8c8c-3468b677908c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 21:02:30.344895 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:02:30.344855    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2" podUID="e7d49a27-d465-49af-8f11-11d586d53a21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 21:02:30.363193 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:02:30.363164    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 21:02:30.363721 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:02:30.363697    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:02:31.348495 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:02:31.348453    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq" podUID="3f2467cf-97ac-40f4-8c8c-3468b677908c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 21:02:40.345048 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:02:40.344958    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2" podUID="e7d49a27-d465-49af-8f11-11d586d53a21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 21:02:40.363409 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:02:40.363376    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 21:02:40.363839 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:02:40.363815    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:02:41.348954 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:02:41.348917    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq" podUID="3f2467cf-97ac-40f4-8c8c-3468b677908c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 21:02:50.344812 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:02:50.344774    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2" podUID="e7d49a27-d465-49af-8f11-11d586d53a21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 21:02:50.363043 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:02:50.363000    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 21:02:50.363471 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:02:50.363442    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:02:50.459371 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:02:50.459343    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq"
Sep 17 21:03:00.344355 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:00.344323    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2"
Sep 17 21:03:00.363383 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:00.363360    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"
Sep 17 21:03:00.363756 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:00.363729    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"
Sep 17 21:03:00.481264 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:00.481237    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq"]
Sep 17 21:03:00.481489 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:00.481466    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq" podUID="3f2467cf-97ac-40f4-8c8c-3468b677908c" containerName="kserve-container" containerID="cri-o://fdd6e92f65fe23ea7b109918511e8b6c227aab244a496b51503ace665d737ec2" gracePeriod=30
Sep 17 21:03:00.481565 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:00.481480    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq" podUID="3f2467cf-97ac-40f4-8c8c-3468b677908c" containerName="kube-rbac-proxy" containerID="cri-o://8f85d941c196626e2e929ad9109f4b4798afcd56b5126ed58306563ae610fecf" gracePeriod=30
Sep 17 21:03:00.696398 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:00.696366    2566 generic.go:356] "Generic (PLEG): container finished" podID="3f2467cf-97ac-40f4-8c8c-3468b677908c" containerID="8f85d941c196626e2e929ad9109f4b4798afcd56b5126ed58306563ae610fecf" exitCode=2
Sep 17 21:03:00.696544 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:00.696402    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq" event={"ID":"3f2467cf-97ac-40f4-8c8c-3468b677908c","Type":"ContainerDied","Data":"8f85d941c196626e2e929ad9109f4b4798afcd56b5126ed58306563ae610fecf"}
Sep 17 21:03:01.344316 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:01.344276    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq" podUID="3f2467cf-97ac-40f4-8c8c-3468b677908c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 17 21:03:02.193489 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:02.193460    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2"]
Sep 17 21:03:02.193896 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:02.193695    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2" podUID="e7d49a27-d465-49af-8f11-11d586d53a21" containerName="kserve-container" containerID="cri-o://c0bd42eca8bc008c533578e3fe50e8430e80a4efbd9339975cd3d18ef4e29c1d" gracePeriod=30
Sep 17 21:03:02.193896 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:02.193742    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2" podUID="e7d49a27-d465-49af-8f11-11d586d53a21" containerName="kube-rbac-proxy" containerID="cri-o://f6d954088c41858a672bd13298c4398a904e8670f4be437ab4e124cae51c3de9" gracePeriod=30
Sep 17 21:03:02.272677 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:02.272646    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv"]
Sep 17 21:03:02.276079 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:02.276059    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv"
Sep 17 21:03:02.287223 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:02.287183    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv"]
Sep 17 21:03:02.393412 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:02.393371    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wvpb\" (UniqueName: \"kubernetes.io/projected/8ffdcb55-f68a-4cbf-941a-0a6c817afc12-kube-api-access-2wvpb\") pod \"isvc-canary-rollback-predictor-97977b6b5-lfppv\" (UID: \"8ffdcb55-f68a-4cbf-941a-0a6c817afc12\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv"
Sep 17 21:03:02.393412 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:02.393420    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ffdcb55-f68a-4cbf-941a-0a6c817afc12-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-97977b6b5-lfppv\" (UID: \"8ffdcb55-f68a-4cbf-941a-0a6c817afc12\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv"
Sep 17 21:03:02.393677 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:02.393515    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ffdcb55-f68a-4cbf-941a-0a6c817afc12-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-97977b6b5-lfppv\" (UID: \"8ffdcb55-f68a-4cbf-941a-0a6c817afc12\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv"
Sep 17 21:03:02.393677 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:02.393604    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ffdcb55-f68a-4cbf-941a-0a6c817afc12-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-lfppv\" (UID: \"8ffdcb55-f68a-4cbf-941a-0a6c817afc12\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv"
Sep 17 21:03:02.393677 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:02.393639    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ffdcb55-f68a-4cbf-941a-0a6c817afc12-cabundle-cert\") pod \"isvc-canary-rollback-predictor-97977b6b5-lfppv\" (UID: \"8ffdcb55-f68a-4cbf-941a-0a6c817afc12\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv"
Sep 17 21:03:02.494099 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:02.494070    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wvpb\" (UniqueName: \"kubernetes.io/projected/8ffdcb55-f68a-4cbf-941a-0a6c817afc12-kube-api-access-2wvpb\") pod \"isvc-canary-rollback-predictor-97977b6b5-lfppv\" (UID: \"8ffdcb55-f68a-4cbf-941a-0a6c817afc12\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv"
Sep 17 21:03:02.494099 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:02.494108    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ffdcb55-f68a-4cbf-941a-0a6c817afc12-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-97977b6b5-lfppv\" (UID: \"8ffdcb55-f68a-4cbf-941a-0a6c817afc12\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv"
Sep 17 21:03:02.494363 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:02.494150    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ffdcb55-f68a-4cbf-941a-0a6c817afc12-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-97977b6b5-lfppv\" (UID: \"8ffdcb55-f68a-4cbf-941a-0a6c817afc12\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv"
Sep 17 21:03:02.494363 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:02.494181    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ffdcb55-f68a-4cbf-941a-0a6c817afc12-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-lfppv\" (UID: \"8ffdcb55-f68a-4cbf-941a-0a6c817afc12\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv"
Sep 17 21:03:02.494363 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:02.494220    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ffdcb55-f68a-4cbf-941a-0a6c817afc12-cabundle-cert\") pod \"isvc-canary-rollback-predictor-97977b6b5-lfppv\" (UID: \"8ffdcb55-f68a-4cbf-941a-0a6c817afc12\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv"
Sep 17 21:03:02.494614 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:02.494582    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ffdcb55-f68a-4cbf-941a-0a6c817afc12-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-97977b6b5-lfppv\" (UID: \"8ffdcb55-f68a-4cbf-941a-0a6c817afc12\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv"
Sep 17 21:03:02.494930 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:02.494905    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ffdcb55-f68a-4cbf-941a-0a6c817afc12-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-97977b6b5-lfppv\" (UID: \"8ffdcb55-f68a-4cbf-941a-0a6c817afc12\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv"
Sep 17 21:03:02.495017 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:02.494976    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ffdcb55-f68a-4cbf-941a-0a6c817afc12-cabundle-cert\") pod \"isvc-canary-rollback-predictor-97977b6b5-lfppv\" (UID: \"8ffdcb55-f68a-4cbf-941a-0a6c817afc12\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv"
Sep 17 21:03:02.496804 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:02.496782    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ffdcb55-f68a-4cbf-941a-0a6c817afc12-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-lfppv\" (UID: \"8ffdcb55-f68a-4cbf-941a-0a6c817afc12\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv"
Sep 17 21:03:02.504858 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:02.504835    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wvpb\" (UniqueName: \"kubernetes.io/projected/8ffdcb55-f68a-4cbf-941a-0a6c817afc12-kube-api-access-2wvpb\") pod \"isvc-canary-rollback-predictor-97977b6b5-lfppv\" (UID: \"8ffdcb55-f68a-4cbf-941a-0a6c817afc12\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv"
Sep 17 21:03:02.585896 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:02.585866    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv"
Sep 17 21:03:02.706507 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:02.706443    2566 generic.go:356] "Generic (PLEG): container finished" podID="e7d49a27-d465-49af-8f11-11d586d53a21" containerID="f6d954088c41858a672bd13298c4398a904e8670f4be437ab4e124cae51c3de9" exitCode=2
Sep 17 21:03:02.706668 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:02.706528    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2" event={"ID":"e7d49a27-d465-49af-8f11-11d586d53a21","Type":"ContainerDied","Data":"f6d954088c41858a672bd13298c4398a904e8670f4be437ab4e124cae51c3de9"}
Sep 17 21:03:02.863175 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:02.863153    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv"]
Sep 17 21:03:02.865712 ip-10-0-138-140 kubenswrapper[2566]: W0917 21:03:02.865684    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ffdcb55_f68a_4cbf_941a_0a6c817afc12.slice/crio-f46179a5937b57f378ab0b0d9dea35817c3335016b8aaec0fb82751f615132b4 WatchSource:0}: Error finding container f46179a5937b57f378ab0b0d9dea35817c3335016b8aaec0fb82751f615132b4: Status 404 returned error can't find the container with id f46179a5937b57f378ab0b0d9dea35817c3335016b8aaec0fb82751f615132b4
Sep 17 21:03:03.712554 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:03.712523    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv" event={"ID":"8ffdcb55-f68a-4cbf-941a-0a6c817afc12","Type":"ContainerStarted","Data":"5683f6cab799adb79d2a0ff9cae065c19f0c741aeffb1d26320b77b588e04182"}
Sep 17 21:03:03.712960 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:03.712579    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv" event={"ID":"8ffdcb55-f68a-4cbf-941a-0a6c817afc12","Type":"ContainerStarted","Data":"f46179a5937b57f378ab0b0d9dea35817c3335016b8aaec0fb82751f615132b4"}
Sep 17 21:03:05.326357 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.326329    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq"
Sep 17 21:03:05.339643 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.339616    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2" podUID="e7d49a27-d465-49af-8f11-11d586d53a21" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 17 21:03:05.419580 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.419555    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f2467cf-97ac-40f4-8c8c-3468b677908c-proxy-tls\") pod \"3f2467cf-97ac-40f4-8c8c-3468b677908c\" (UID: \"3f2467cf-97ac-40f4-8c8c-3468b677908c\") "
Sep 17 21:03:05.419723 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.419590    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f2467cf-97ac-40f4-8c8c-3468b677908c-kserve-provision-location\") pod \"3f2467cf-97ac-40f4-8c8c-3468b677908c\" (UID: \"3f2467cf-97ac-40f4-8c8c-3468b677908c\") "
Sep 17 21:03:05.419723 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.419626    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f2467cf-97ac-40f4-8c8c-3468b677908c-kube-rbac-proxy-sar-config\") pod \"3f2467cf-97ac-40f4-8c8c-3468b677908c\" (UID: \"3f2467cf-97ac-40f4-8c8c-3468b677908c\") "
Sep 17 21:03:05.419723 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.419675    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfk9x\" (UniqueName: \"kubernetes.io/projected/3f2467cf-97ac-40f4-8c8c-3468b677908c-kube-api-access-pfk9x\") pod \"3f2467cf-97ac-40f4-8c8c-3468b677908c\" (UID: \"3f2467cf-97ac-40f4-8c8c-3468b677908c\") "
Sep 17 21:03:05.419723 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.419699    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f2467cf-97ac-40f4-8c8c-3468b677908c-cabundle-cert\") pod \"3f2467cf-97ac-40f4-8c8c-3468b677908c\" (UID: \"3f2467cf-97ac-40f4-8c8c-3468b677908c\") "
Sep 17 21:03:05.419946 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.419881    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2467cf-97ac-40f4-8c8c-3468b677908c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3f2467cf-97ac-40f4-8c8c-3468b677908c" (UID: "3f2467cf-97ac-40f4-8c8c-3468b677908c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:03:05.420032 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.420000    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f2467cf-97ac-40f4-8c8c-3468b677908c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3f2467cf-97ac-40f4-8c8c-3468b677908c" (UID: "3f2467cf-97ac-40f4-8c8c-3468b677908c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:03:05.420160 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.420136    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f2467cf-97ac-40f4-8c8c-3468b677908c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3f2467cf-97ac-40f4-8c8c-3468b677908c" (UID: "3f2467cf-97ac-40f4-8c8c-3468b677908c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:03:05.421833 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.421809    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f2467cf-97ac-40f4-8c8c-3468b677908c-kube-api-access-pfk9x" (OuterVolumeSpecName: "kube-api-access-pfk9x") pod "3f2467cf-97ac-40f4-8c8c-3468b677908c" (UID: "3f2467cf-97ac-40f4-8c8c-3468b677908c"). InnerVolumeSpecName "kube-api-access-pfk9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:03:05.422141 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.421834    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f2467cf-97ac-40f4-8c8c-3468b677908c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3f2467cf-97ac-40f4-8c8c-3468b677908c" (UID: "3f2467cf-97ac-40f4-8c8c-3468b677908c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:03:05.520806 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.520778    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfk9x\" (UniqueName: \"kubernetes.io/projected/3f2467cf-97ac-40f4-8c8c-3468b677908c-kube-api-access-pfk9x\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:03:05.520806 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.520799    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f2467cf-97ac-40f4-8c8c-3468b677908c-cabundle-cert\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:03:05.520806 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.520809    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f2467cf-97ac-40f4-8c8c-3468b677908c-proxy-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:03:05.521082 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.520816    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f2467cf-97ac-40f4-8c8c-3468b677908c-kserve-provision-location\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:03:05.521082 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.520824    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f2467cf-97ac-40f4-8c8c-3468b677908c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:03:05.722686 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.722647    2566 generic.go:356] "Generic (PLEG): container finished" podID="3f2467cf-97ac-40f4-8c8c-3468b677908c" containerID="fdd6e92f65fe23ea7b109918511e8b6c227aab244a496b51503ace665d737ec2" exitCode=0
Sep 17 21:03:05.722850 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.722706    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq"
Sep 17 21:03:05.722850 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.722727    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq" event={"ID":"3f2467cf-97ac-40f4-8c8c-3468b677908c","Type":"ContainerDied","Data":"fdd6e92f65fe23ea7b109918511e8b6c227aab244a496b51503ace665d737ec2"}
Sep 17 21:03:05.722850 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.722793    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq" event={"ID":"3f2467cf-97ac-40f4-8c8c-3468b677908c","Type":"ContainerDied","Data":"5a4970d55d3de54a23b1cf8a9aa094f4c88518e98a1e7f77eb6ccf3f38bce6df"}
Sep 17 21:03:05.722850 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.722832    2566 scope.go:117] "RemoveContainer" containerID="8f85d941c196626e2e929ad9109f4b4798afcd56b5126ed58306563ae610fecf"
Sep 17 21:03:05.731137 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.731118    2566 scope.go:117] "RemoveContainer" containerID="fdd6e92f65fe23ea7b109918511e8b6c227aab244a496b51503ace665d737ec2"
Sep 17 21:03:05.738252 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.738234    2566 scope.go:117] "RemoveContainer" containerID="6ba3d87ff194470158ead06f862c8cc7effb844bfe690747a4c898af59cd3fc0"
Sep 17 21:03:05.745147 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.745121    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq"]
Sep 17 21:03:05.745874 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.745858    2566 scope.go:117] "RemoveContainer" containerID="8f85d941c196626e2e929ad9109f4b4798afcd56b5126ed58306563ae610fecf"
Sep 17 21:03:05.746140 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:03:05.746121    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f85d941c196626e2e929ad9109f4b4798afcd56b5126ed58306563ae610fecf\": container with ID starting with 8f85d941c196626e2e929ad9109f4b4798afcd56b5126ed58306563ae610fecf not found: ID does not exist" containerID="8f85d941c196626e2e929ad9109f4b4798afcd56b5126ed58306563ae610fecf"
Sep 17 21:03:05.746193 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.746149    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f85d941c196626e2e929ad9109f4b4798afcd56b5126ed58306563ae610fecf"} err="failed to get container status \"8f85d941c196626e2e929ad9109f4b4798afcd56b5126ed58306563ae610fecf\": rpc error: code = NotFound desc = could not find container \"8f85d941c196626e2e929ad9109f4b4798afcd56b5126ed58306563ae610fecf\": container with ID starting with 8f85d941c196626e2e929ad9109f4b4798afcd56b5126ed58306563ae610fecf not found: ID does not exist"
Sep 17 21:03:05.746193 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.746166    2566 scope.go:117] "RemoveContainer" containerID="fdd6e92f65fe23ea7b109918511e8b6c227aab244a496b51503ace665d737ec2"
Sep 17 21:03:05.746463 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:03:05.746445    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdd6e92f65fe23ea7b109918511e8b6c227aab244a496b51503ace665d737ec2\": container with ID starting with fdd6e92f65fe23ea7b109918511e8b6c227aab244a496b51503ace665d737ec2 not found: ID does not exist" containerID="fdd6e92f65fe23ea7b109918511e8b6c227aab244a496b51503ace665d737ec2"
Sep 17 21:03:05.746506 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.746469    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdd6e92f65fe23ea7b109918511e8b6c227aab244a496b51503ace665d737ec2"} err="failed to get container status \"fdd6e92f65fe23ea7b109918511e8b6c227aab244a496b51503ace665d737ec2\": rpc error: code = NotFound desc = could not find container \"fdd6e92f65fe23ea7b109918511e8b6c227aab244a496b51503ace665d737ec2\": container with ID starting with fdd6e92f65fe23ea7b109918511e8b6c227aab244a496b51503ace665d737ec2 not found: ID does not exist"
Sep 17 21:03:05.746506 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.746487    2566 scope.go:117] "RemoveContainer" containerID="6ba3d87ff194470158ead06f862c8cc7effb844bfe690747a4c898af59cd3fc0"
Sep 17 21:03:05.746727 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:03:05.746709    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ba3d87ff194470158ead06f862c8cc7effb844bfe690747a4c898af59cd3fc0\": container with ID starting with 6ba3d87ff194470158ead06f862c8cc7effb844bfe690747a4c898af59cd3fc0 not found: ID does not exist" containerID="6ba3d87ff194470158ead06f862c8cc7effb844bfe690747a4c898af59cd3fc0"
Sep 17 21:03:05.746781 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.746734    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ba3d87ff194470158ead06f862c8cc7effb844bfe690747a4c898af59cd3fc0"} err="failed to get container status \"6ba3d87ff194470158ead06f862c8cc7effb844bfe690747a4c898af59cd3fc0\": rpc error: code = NotFound desc = could not find container \"6ba3d87ff194470158ead06f862c8cc7effb844bfe690747a4c898af59cd3fc0\": container with ID starting with 6ba3d87ff194470158ead06f862c8cc7effb844bfe690747a4c898af59cd3fc0 not found: ID does not exist"
Sep 17 21:03:05.750993 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:05.750973    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4dlsq"]
Sep 17 21:03:06.459505 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:06.459475    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f2467cf-97ac-40f4-8c8c-3468b677908c" path="/var/lib/kubelet/pods/3f2467cf-97ac-40f4-8c8c-3468b677908c/volumes"
Sep 17 21:03:06.935826 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:06.935805    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2"
Sep 17 21:03:07.033606 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.033576    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7d49a27-d465-49af-8f11-11d586d53a21-cabundle-cert\") pod \"e7d49a27-d465-49af-8f11-11d586d53a21\" (UID: \"e7d49a27-d465-49af-8f11-11d586d53a21\") "
Sep 17 21:03:07.033777 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.033619    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7d49a27-d465-49af-8f11-11d586d53a21-proxy-tls\") pod \"e7d49a27-d465-49af-8f11-11d586d53a21\" (UID: \"e7d49a27-d465-49af-8f11-11d586d53a21\") "
Sep 17 21:03:07.033777 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.033672    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7d49a27-d465-49af-8f11-11d586d53a21-kserve-provision-location\") pod \"e7d49a27-d465-49af-8f11-11d586d53a21\" (UID: \"e7d49a27-d465-49af-8f11-11d586d53a21\") "
Sep 17 21:03:07.033777 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.033729    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7d49a27-d465-49af-8f11-11d586d53a21-kube-rbac-proxy-sar-config\") pod \"e7d49a27-d465-49af-8f11-11d586d53a21\" (UID: \"e7d49a27-d465-49af-8f11-11d586d53a21\") "
Sep 17 21:03:07.033777 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.033753    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9sjl\" (UniqueName: \"kubernetes.io/projected/e7d49a27-d465-49af-8f11-11d586d53a21-kube-api-access-x9sjl\") pod \"e7d49a27-d465-49af-8f11-11d586d53a21\" (UID: \"e7d49a27-d465-49af-8f11-11d586d53a21\") "
Sep 17 21:03:07.034008 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.033981    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7d49a27-d465-49af-8f11-11d586d53a21-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e7d49a27-d465-49af-8f11-11d586d53a21" (UID: "e7d49a27-d465-49af-8f11-11d586d53a21"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:03:07.034069 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.034022    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7d49a27-d465-49af-8f11-11d586d53a21-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e7d49a27-d465-49af-8f11-11d586d53a21" (UID: "e7d49a27-d465-49af-8f11-11d586d53a21"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:03:07.034123 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.034101    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7d49a27-d465-49af-8f11-11d586d53a21-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e7d49a27-d465-49af-8f11-11d586d53a21" (UID: "e7d49a27-d465-49af-8f11-11d586d53a21"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:03:07.035887 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.035862    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7d49a27-d465-49af-8f11-11d586d53a21-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e7d49a27-d465-49af-8f11-11d586d53a21" (UID: "e7d49a27-d465-49af-8f11-11d586d53a21"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:03:07.036000 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.035895    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d49a27-d465-49af-8f11-11d586d53a21-kube-api-access-x9sjl" (OuterVolumeSpecName: "kube-api-access-x9sjl") pod "e7d49a27-d465-49af-8f11-11d586d53a21" (UID: "e7d49a27-d465-49af-8f11-11d586d53a21"). InnerVolumeSpecName "kube-api-access-x9sjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:03:07.134626 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.134550    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7d49a27-d465-49af-8f11-11d586d53a21-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:03:07.134626 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.134579    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9sjl\" (UniqueName: \"kubernetes.io/projected/e7d49a27-d465-49af-8f11-11d586d53a21-kube-api-access-x9sjl\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:03:07.134626 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.134589    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7d49a27-d465-49af-8f11-11d586d53a21-cabundle-cert\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:03:07.134626 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.134600    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7d49a27-d465-49af-8f11-11d586d53a21-proxy-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:03:07.134626 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.134620    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7d49a27-d465-49af-8f11-11d586d53a21-kserve-provision-location\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:03:07.733727 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.733691    2566 generic.go:356] "Generic (PLEG): container finished" podID="e7d49a27-d465-49af-8f11-11d586d53a21" containerID="c0bd42eca8bc008c533578e3fe50e8430e80a4efbd9339975cd3d18ef4e29c1d" exitCode=0
Sep 17 21:03:07.734155 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.733745    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2"
Sep 17 21:03:07.734155 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.733771    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2" event={"ID":"e7d49a27-d465-49af-8f11-11d586d53a21","Type":"ContainerDied","Data":"c0bd42eca8bc008c533578e3fe50e8430e80a4efbd9339975cd3d18ef4e29c1d"}
Sep 17 21:03:07.734155 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.733829    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2" event={"ID":"e7d49a27-d465-49af-8f11-11d586d53a21","Type":"ContainerDied","Data":"73868b5686fe0d01fb7312abd37285c652fdcc8c38ef0971bd943f42dc416e8a"}
Sep 17 21:03:07.734155 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.733867    2566 scope.go:117] "RemoveContainer" containerID="f6d954088c41858a672bd13298c4398a904e8670f4be437ab4e124cae51c3de9"
Sep 17 21:03:07.735450 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.735427    2566 generic.go:356] "Generic (PLEG): container finished" podID="8ffdcb55-f68a-4cbf-941a-0a6c817afc12" containerID="5683f6cab799adb79d2a0ff9cae065c19f0c741aeffb1d26320b77b588e04182" exitCode=0
Sep 17 21:03:07.735566 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.735492    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv" event={"ID":"8ffdcb55-f68a-4cbf-941a-0a6c817afc12","Type":"ContainerDied","Data":"5683f6cab799adb79d2a0ff9cae065c19f0c741aeffb1d26320b77b588e04182"}
Sep 17 21:03:07.743046 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.743026    2566 scope.go:117] "RemoveContainer" containerID="c0bd42eca8bc008c533578e3fe50e8430e80a4efbd9339975cd3d18ef4e29c1d"
Sep 17 21:03:07.750559 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.750538    2566 scope.go:117] "RemoveContainer" containerID="a0583feeab1e4e1cc75c148eaaf698e3394aa211f02c93774371ae8e2655ae4f"
Sep 17 21:03:07.761183 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.759874    2566 scope.go:117] "RemoveContainer" containerID="f6d954088c41858a672bd13298c4398a904e8670f4be437ab4e124cae51c3de9"
Sep 17 21:03:07.761614 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:03:07.761592    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6d954088c41858a672bd13298c4398a904e8670f4be437ab4e124cae51c3de9\": container with ID starting with f6d954088c41858a672bd13298c4398a904e8670f4be437ab4e124cae51c3de9 not found: ID does not exist" containerID="f6d954088c41858a672bd13298c4398a904e8670f4be437ab4e124cae51c3de9"
Sep 17 21:03:07.761671 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.761618    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d954088c41858a672bd13298c4398a904e8670f4be437ab4e124cae51c3de9"} err="failed to get container status \"f6d954088c41858a672bd13298c4398a904e8670f4be437ab4e124cae51c3de9\": rpc error: code = NotFound desc = could not find container \"f6d954088c41858a672bd13298c4398a904e8670f4be437ab4e124cae51c3de9\": container with ID starting with f6d954088c41858a672bd13298c4398a904e8670f4be437ab4e124cae51c3de9 not found: ID does not exist"
Sep 17 21:03:07.761671 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.761636    2566 scope.go:117] "RemoveContainer" containerID="c0bd42eca8bc008c533578e3fe50e8430e80a4efbd9339975cd3d18ef4e29c1d"
Sep 17 21:03:07.761924 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:03:07.761910    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0bd42eca8bc008c533578e3fe50e8430e80a4efbd9339975cd3d18ef4e29c1d\": container with ID starting with c0bd42eca8bc008c533578e3fe50e8430e80a4efbd9339975cd3d18ef4e29c1d not found: ID does not exist" containerID="c0bd42eca8bc008c533578e3fe50e8430e80a4efbd9339975cd3d18ef4e29c1d"
Sep 17 21:03:07.761982 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.761927    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0bd42eca8bc008c533578e3fe50e8430e80a4efbd9339975cd3d18ef4e29c1d"} err="failed to get container status \"c0bd42eca8bc008c533578e3fe50e8430e80a4efbd9339975cd3d18ef4e29c1d\": rpc error: code = NotFound desc = could not find container \"c0bd42eca8bc008c533578e3fe50e8430e80a4efbd9339975cd3d18ef4e29c1d\": container with ID starting with c0bd42eca8bc008c533578e3fe50e8430e80a4efbd9339975cd3d18ef4e29c1d not found: ID does not exist"
Sep 17 21:03:07.761982 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.761938    2566 scope.go:117] "RemoveContainer" containerID="a0583feeab1e4e1cc75c148eaaf698e3394aa211f02c93774371ae8e2655ae4f"
Sep 17 21:03:07.762201 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:03:07.762183    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0583feeab1e4e1cc75c148eaaf698e3394aa211f02c93774371ae8e2655ae4f\": container with ID starting with a0583feeab1e4e1cc75c148eaaf698e3394aa211f02c93774371ae8e2655ae4f not found: ID does not exist" containerID="a0583feeab1e4e1cc75c148eaaf698e3394aa211f02c93774371ae8e2655ae4f"
Sep 17 21:03:07.762283 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.762227    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0583feeab1e4e1cc75c148eaaf698e3394aa211f02c93774371ae8e2655ae4f"} err="failed to get container status \"a0583feeab1e4e1cc75c148eaaf698e3394aa211f02c93774371ae8e2655ae4f\": rpc error: code = NotFound desc = could not find container \"a0583feeab1e4e1cc75c148eaaf698e3394aa211f02c93774371ae8e2655ae4f\": container with ID starting with a0583feeab1e4e1cc75c148eaaf698e3394aa211f02c93774371ae8e2655ae4f not found: ID does not exist"
Sep 17 21:03:07.769171 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.769151    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2"]
Sep 17 21:03:07.783591 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:07.783570    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-czkk2"]
Sep 17 21:03:08.458902 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:08.458868    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7d49a27-d465-49af-8f11-11d586d53a21" path="/var/lib/kubelet/pods/e7d49a27-d465-49af-8f11-11d586d53a21/volumes"
Sep 17 21:03:08.742368 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:08.742338    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv" event={"ID":"8ffdcb55-f68a-4cbf-941a-0a6c817afc12","Type":"ContainerStarted","Data":"c920df1e6eef61e2b658c7ba887c96e92d478b874c90b6e4bdfddc215ffb4222"}
Sep 17 21:03:08.742752 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:08.742413    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv" event={"ID":"8ffdcb55-f68a-4cbf-941a-0a6c817afc12","Type":"ContainerStarted","Data":"64bc584f3eab94ac0d3b36cab01728b8aa6a8bd8402f2acaf4e2e8226e9806ae"}
Sep 17 21:03:08.742752 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:08.742607    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv"
Sep 17 21:03:08.764778 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:08.764731    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv" podStartSLOduration=2.873089139 podStartE2EDuration="6.764696957s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.891607818s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:03:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:03:08.762473507 +0000 UTC m=+830.373206728" watchObservedRunningTime="2026-09-17 21:03:08.764696957 +0000 UTC m=+830.375430184"
Sep 17 21:03:09.746935 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:09.746905    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv"
Sep 17 21:03:09.748089 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:09.748062    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv" podUID="8ffdcb55-f68a-4cbf-941a-0a6c817afc12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 21:03:10.750442 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:10.750394    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv" podUID="8ffdcb55-f68a-4cbf-941a-0a6c817afc12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 21:03:12.747122 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:12.747088    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"]
Sep 17 21:03:12.747529 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:12.747349    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="kserve-container" containerID="cri-o://38374de9970d8416223fae0074418b135a2f895fce2f1f0d7c7c678f2f5b457d" gracePeriod=30
Sep 17 21:03:12.747529 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:12.747368    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="kube-rbac-proxy" containerID="cri-o://291a59fd04f0a1ff0f1e49d639e48351af2b1b2f91fd90a109f99fe4468c5b1a" gracePeriod=30
Sep 17 21:03:12.747529 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:12.747379    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="agent" containerID="cri-o://404401654b48b62c7b9045f9df573666c68d292204732f05ca6af63f995e3cce" gracePeriod=30
Sep 17 21:03:13.765673 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:13.765641    2566 generic.go:356] "Generic (PLEG): container finished" podID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerID="291a59fd04f0a1ff0f1e49d639e48351af2b1b2f91fd90a109f99fe4468c5b1a" exitCode=2
Sep 17 21:03:13.766045 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:13.765716    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" event={"ID":"6a308e40-a9f1-4a70-aae9-eca1b7b30d37","Type":"ContainerDied","Data":"291a59fd04f0a1ff0f1e49d639e48351af2b1b2f91fd90a109f99fe4468c5b1a"}
Sep 17 21:03:14.358993 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:14.358952    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 17 21:03:15.755564 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:15.755529    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv"
Sep 17 21:03:15.756035 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:15.756009    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv" podUID="8ffdcb55-f68a-4cbf-941a-0a6c817afc12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 21:03:17.784077 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:17.784035    2566 generic.go:356] "Generic (PLEG): container finished" podID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerID="38374de9970d8416223fae0074418b135a2f895fce2f1f0d7c7c678f2f5b457d" exitCode=0
Sep 17 21:03:17.784443 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:17.784090    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" event={"ID":"6a308e40-a9f1-4a70-aae9-eca1b7b30d37","Type":"ContainerDied","Data":"38374de9970d8416223fae0074418b135a2f895fce2f1f0d7c7c678f2f5b457d"}
Sep 17 21:03:19.358858 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:19.358825    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 17 21:03:20.363537 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:20.363500    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 21:03:20.364130 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:20.364104    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:03:24.358616 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:24.358572    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 17 21:03:24.358987 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:24.358712    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"
Sep 17 21:03:25.756607 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:25.756573    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv" podUID="8ffdcb55-f68a-4cbf-941a-0a6c817afc12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 21:03:29.359433 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:29.359390    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 17 21:03:30.363460 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:30.363421    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 21:03:30.364130 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:30.364106    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:03:34.359109 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:34.359068    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 17 21:03:35.756936 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:35.756896    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv" podUID="8ffdcb55-f68a-4cbf-941a-0a6c817afc12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 21:03:39.358779 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:39.358739    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 17 21:03:40.363094 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:40.363056    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 21:03:40.363486 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:40.363230    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"
Sep 17 21:03:40.364650 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:40.364625    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:03:40.364740 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:40.364728    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"
Sep 17 21:03:42.886047 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:42.886021    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"
Sep 17 21:03:42.887246 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:42.887204    2566 generic.go:356] "Generic (PLEG): container finished" podID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerID="404401654b48b62c7b9045f9df573666c68d292204732f05ca6af63f995e3cce" exitCode=0
Sep 17 21:03:42.887350 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:42.887246    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" event={"ID":"6a308e40-a9f1-4a70-aae9-eca1b7b30d37","Type":"ContainerDied","Data":"404401654b48b62c7b9045f9df573666c68d292204732f05ca6af63f995e3cce"}
Sep 17 21:03:42.887350 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:42.887312    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl" event={"ID":"6a308e40-a9f1-4a70-aae9-eca1b7b30d37","Type":"ContainerDied","Data":"110fdb66c4b7eeb167c47c6f644b9372eb8feb2f251931a3c8c4025b7ce96330"}
Sep 17 21:03:42.887350 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:42.887343    2566 scope.go:117] "RemoveContainer" containerID="404401654b48b62c7b9045f9df573666c68d292204732f05ca6af63f995e3cce"
Sep 17 21:03:42.895000 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:42.894982    2566 scope.go:117] "RemoveContainer" containerID="291a59fd04f0a1ff0f1e49d639e48351af2b1b2f91fd90a109f99fe4468c5b1a"
Sep 17 21:03:42.902666 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:42.902652    2566 scope.go:117] "RemoveContainer" containerID="38374de9970d8416223fae0074418b135a2f895fce2f1f0d7c7c678f2f5b457d"
Sep 17 21:03:42.909364 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:42.909350    2566 scope.go:117] "RemoveContainer" containerID="8a79d8175f9bc2701536d22ca4191ac3466a7c7b006075d8cbfbf9f04f301c45"
Sep 17 21:03:42.911042 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:42.911024    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jjmf\" (UniqueName: \"kubernetes.io/projected/6a308e40-a9f1-4a70-aae9-eca1b7b30d37-kube-api-access-9jjmf\") pod \"6a308e40-a9f1-4a70-aae9-eca1b7b30d37\" (UID: \"6a308e40-a9f1-4a70-aae9-eca1b7b30d37\") "
Sep 17 21:03:42.911130 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:42.911067    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a308e40-a9f1-4a70-aae9-eca1b7b30d37-proxy-tls\") pod \"6a308e40-a9f1-4a70-aae9-eca1b7b30d37\" (UID: \"6a308e40-a9f1-4a70-aae9-eca1b7b30d37\") "
Sep 17 21:03:42.911130 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:42.911099    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a308e40-a9f1-4a70-aae9-eca1b7b30d37-cabundle-cert\") pod \"6a308e40-a9f1-4a70-aae9-eca1b7b30d37\" (UID: \"6a308e40-a9f1-4a70-aae9-eca1b7b30d37\") "
Sep 17 21:03:42.911130 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:42.911125    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a308e40-a9f1-4a70-aae9-eca1b7b30d37-kserve-provision-location\") pod \"6a308e40-a9f1-4a70-aae9-eca1b7b30d37\" (UID: \"6a308e40-a9f1-4a70-aae9-eca1b7b30d37\") "
Sep 17 21:03:42.911845 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:42.911153    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a308e40-a9f1-4a70-aae9-eca1b7b30d37-kube-rbac-proxy-sar-config\") pod \"6a308e40-a9f1-4a70-aae9-eca1b7b30d37\" (UID: \"6a308e40-a9f1-4a70-aae9-eca1b7b30d37\") "
Sep 17 21:03:42.911845 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:42.911565    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a308e40-a9f1-4a70-aae9-eca1b7b30d37-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6a308e40-a9f1-4a70-aae9-eca1b7b30d37" (UID: "6a308e40-a9f1-4a70-aae9-eca1b7b30d37"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:03:42.911845 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:42.911607    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a308e40-a9f1-4a70-aae9-eca1b7b30d37-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6a308e40-a9f1-4a70-aae9-eca1b7b30d37" (UID: "6a308e40-a9f1-4a70-aae9-eca1b7b30d37"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:03:42.912424 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:42.912388    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a308e40-a9f1-4a70-aae9-eca1b7b30d37-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6a308e40-a9f1-4a70-aae9-eca1b7b30d37" (UID: "6a308e40-a9f1-4a70-aae9-eca1b7b30d37"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:03:42.913550 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:42.913526    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a308e40-a9f1-4a70-aae9-eca1b7b30d37-kube-api-access-9jjmf" (OuterVolumeSpecName: "kube-api-access-9jjmf") pod "6a308e40-a9f1-4a70-aae9-eca1b7b30d37" (UID: "6a308e40-a9f1-4a70-aae9-eca1b7b30d37"). InnerVolumeSpecName "kube-api-access-9jjmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:03:42.913884 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:42.913857    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a308e40-a9f1-4a70-aae9-eca1b7b30d37-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a308e40-a9f1-4a70-aae9-eca1b7b30d37" (UID: "6a308e40-a9f1-4a70-aae9-eca1b7b30d37"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:03:42.918059 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:42.918036    2566 scope.go:117] "RemoveContainer" containerID="404401654b48b62c7b9045f9df573666c68d292204732f05ca6af63f995e3cce"
Sep 17 21:03:42.918339 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:03:42.918313    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"404401654b48b62c7b9045f9df573666c68d292204732f05ca6af63f995e3cce\": container with ID starting with 404401654b48b62c7b9045f9df573666c68d292204732f05ca6af63f995e3cce not found: ID does not exist" containerID="404401654b48b62c7b9045f9df573666c68d292204732f05ca6af63f995e3cce"
Sep 17 21:03:42.918414 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:42.918351    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"404401654b48b62c7b9045f9df573666c68d292204732f05ca6af63f995e3cce"} err="failed to get container status \"404401654b48b62c7b9045f9df573666c68d292204732f05ca6af63f995e3cce\": rpc error: code = NotFound desc = could not find container \"404401654b48b62c7b9045f9df573666c68d292204732f05ca6af63f995e3cce\": container with ID starting with 404401654b48b62c7b9045f9df573666c68d292204732f05ca6af63f995e3cce not found: ID does not exist"
Sep 17 21:03:42.918414 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:42.918374    2566 scope.go:117] "RemoveContainer" containerID="291a59fd04f0a1ff0f1e49d639e48351af2b1b2f91fd90a109f99fe4468c5b1a"
Sep 17 21:03:42.918659 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:03:42.918642    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"291a59fd04f0a1ff0f1e49d639e48351af2b1b2f91fd90a109f99fe4468c5b1a\": container with ID starting with 291a59fd04f0a1ff0f1e49d639e48351af2b1b2f91fd90a109f99fe4468c5b1a not found: ID does not exist" containerID="291a59fd04f0a1ff0f1e49d639e48351af2b1b2f91fd90a109f99fe4468c5b1a"
Sep 17 21:03:42.918712 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:42.918665    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"291a59fd04f0a1ff0f1e49d639e48351af2b1b2f91fd90a109f99fe4468c5b1a"} err="failed to get container status \"291a59fd04f0a1ff0f1e49d639e48351af2b1b2f91fd90a109f99fe4468c5b1a\": rpc error: code = NotFound desc = could not find container \"291a59fd04f0a1ff0f1e49d639e48351af2b1b2f91fd90a109f99fe4468c5b1a\": container with ID starting with 291a59fd04f0a1ff0f1e49d639e48351af2b1b2f91fd90a109f99fe4468c5b1a not found: ID does not exist"
Sep 17 21:03:42.918712 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:42.918680    2566 scope.go:117] "RemoveContainer" containerID="38374de9970d8416223fae0074418b135a2f895fce2f1f0d7c7c678f2f5b457d"
Sep 17 21:03:42.919104 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:03:42.918948    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38374de9970d8416223fae0074418b135a2f895fce2f1f0d7c7c678f2f5b457d\": container with ID starting with 38374de9970d8416223fae0074418b135a2f895fce2f1f0d7c7c678f2f5b457d not found: ID does not exist" containerID="38374de9970d8416223fae0074418b135a2f895fce2f1f0d7c7c678f2f5b457d"
Sep 17 21:03:42.919104 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:42.918979    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38374de9970d8416223fae0074418b135a2f895fce2f1f0d7c7c678f2f5b457d"} err="failed to get container status \"38374de9970d8416223fae0074418b135a2f895fce2f1f0d7c7c678f2f5b457d\": rpc error: code = NotFound desc = could not find container \"38374de9970d8416223fae0074418b135a2f895fce2f1f0d7c7c678f2f5b457d\": container with ID starting with 38374de9970d8416223fae0074418b135a2f895fce2f1f0d7c7c678f2f5b457d not found: ID does not exist"
Sep 17 21:03:42.919104 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:42.919000    2566 scope.go:117] "RemoveContainer" containerID="8a79d8175f9bc2701536d22ca4191ac3466a7c7b006075d8cbfbf9f04f301c45"
Sep 17 21:03:42.919389 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:03:42.919361    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a79d8175f9bc2701536d22ca4191ac3466a7c7b006075d8cbfbf9f04f301c45\": container with ID starting with 8a79d8175f9bc2701536d22ca4191ac3466a7c7b006075d8cbfbf9f04f301c45 not found: ID does not exist" containerID="8a79d8175f9bc2701536d22ca4191ac3466a7c7b006075d8cbfbf9f04f301c45"
Sep 17 21:03:42.919443 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:42.919386    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a79d8175f9bc2701536d22ca4191ac3466a7c7b006075d8cbfbf9f04f301c45"} err="failed to get container status \"8a79d8175f9bc2701536d22ca4191ac3466a7c7b006075d8cbfbf9f04f301c45\": rpc error: code = NotFound desc = could not find container \"8a79d8175f9bc2701536d22ca4191ac3466a7c7b006075d8cbfbf9f04f301c45\": container with ID starting with 8a79d8175f9bc2701536d22ca4191ac3466a7c7b006075d8cbfbf9f04f301c45 not found: ID does not exist"
Sep 17 21:03:43.012708 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:43.012648    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a308e40-a9f1-4a70-aae9-eca1b7b30d37-proxy-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:03:43.012708 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:43.012670    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a308e40-a9f1-4a70-aae9-eca1b7b30d37-cabundle-cert\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:03:43.012708 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:43.012680    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a308e40-a9f1-4a70-aae9-eca1b7b30d37-kserve-provision-location\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:03:43.012708 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:43.012689    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a308e40-a9f1-4a70-aae9-eca1b7b30d37-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:03:43.012708 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:43.012697    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jjmf\" (UniqueName: \"kubernetes.io/projected/6a308e40-a9f1-4a70-aae9-eca1b7b30d37-kube-api-access-9jjmf\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:03:43.892978 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:43.892948    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"
Sep 17 21:03:43.917807 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:43.917781    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"]
Sep 17 21:03:43.923756 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:43.923731    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-89dfa-predictor-76d8855cfb-pslrl"]
Sep 17 21:03:44.459643 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:44.459611    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" path="/var/lib/kubelet/pods/6a308e40-a9f1-4a70-aae9-eca1b7b30d37/volumes"
Sep 17 21:03:45.756855 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:45.756817    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv" podUID="8ffdcb55-f68a-4cbf-941a-0a6c817afc12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 21:03:55.756457 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:03:55.756420    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv" podUID="8ffdcb55-f68a-4cbf-941a-0a6c817afc12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 21:04:05.756240 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:05.756151    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv" podUID="8ffdcb55-f68a-4cbf-941a-0a6c817afc12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 21:04:15.756264 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:15.756229    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv" podUID="8ffdcb55-f68a-4cbf-941a-0a6c817afc12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 21:04:20.419040 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:20.419007    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t2rlv_ed6472a6-7979-43b0-a549-7ca5c5d8fb0f/console-operator/1.log"
Sep 17 21:04:20.419597 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:20.419086    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t2rlv_ed6472a6-7979-43b0-a549-7ca5c5d8fb0f/console-operator/1.log"
Sep 17 21:04:20.424733 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:20.424713    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnqx7_402f55a4-24b6-47c6-b719-9c6a5b473434/ovn-acl-logging/0.log"
Sep 17 21:04:20.424733 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:20.424726    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnqx7_402f55a4-24b6-47c6-b719-9c6a5b473434/ovn-acl-logging/0.log"
Sep 17 21:04:21.455420 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:21.455387    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv"
Sep 17 21:04:22.318260 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:22.318202    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv"]
Sep 17 21:04:23.046509 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:23.046447    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv" podUID="8ffdcb55-f68a-4cbf-941a-0a6c817afc12" containerName="kserve-container" containerID="cri-o://64bc584f3eab94ac0d3b36cab01728b8aa6a8bd8402f2acaf4e2e8226e9806ae" gracePeriod=30
Sep 17 21:04:23.046965 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:23.046512    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv" podUID="8ffdcb55-f68a-4cbf-941a-0a6c817afc12" containerName="kube-rbac-proxy" containerID="cri-o://c920df1e6eef61e2b658c7ba887c96e92d478b874c90b6e4bdfddc215ffb4222" gracePeriod=30
Sep 17 21:04:24.052885 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:24.052852    2566 generic.go:356] "Generic (PLEG): container finished" podID="8ffdcb55-f68a-4cbf-941a-0a6c817afc12" containerID="c920df1e6eef61e2b658c7ba887c96e92d478b874c90b6e4bdfddc215ffb4222" exitCode=2
Sep 17 21:04:24.052885 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:24.052887    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv" event={"ID":"8ffdcb55-f68a-4cbf-941a-0a6c817afc12","Type":"ContainerDied","Data":"c920df1e6eef61e2b658c7ba887c96e92d478b874c90b6e4bdfddc215ffb4222"}
Sep 17 21:04:25.750960 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:25.750912    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv" podUID="8ffdcb55-f68a-4cbf-941a-0a6c817afc12" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 17 21:04:27.584277 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.584248    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv"
Sep 17 21:04:27.661634 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.661562    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd"]
Sep 17 21:04:27.661854 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.661842    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7d49a27-d465-49af-8f11-11d586d53a21" containerName="storage-initializer"
Sep 17 21:04:27.661895 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.661855    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7d49a27-d465-49af-8f11-11d586d53a21" containerName="storage-initializer"
Sep 17 21:04:27.661895 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.661864    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f2467cf-97ac-40f4-8c8c-3468b677908c" containerName="kube-rbac-proxy"
Sep 17 21:04:27.661895 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.661870    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f2467cf-97ac-40f4-8c8c-3468b677908c" containerName="kube-rbac-proxy"
Sep 17 21:04:27.661895 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.661879    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="kserve-container"
Sep 17 21:04:27.661895 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.661886    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="kserve-container"
Sep 17 21:04:27.661895 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.661896    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f2467cf-97ac-40f4-8c8c-3468b677908c" containerName="storage-initializer"
Sep 17 21:04:27.662087 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.661902    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f2467cf-97ac-40f4-8c8c-3468b677908c" containerName="storage-initializer"
Sep 17 21:04:27.662087 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.661909    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7d49a27-d465-49af-8f11-11d586d53a21" containerName="kube-rbac-proxy"
Sep 17 21:04:27.662087 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.661933    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7d49a27-d465-49af-8f11-11d586d53a21" containerName="kube-rbac-proxy"
Sep 17 21:04:27.662087 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.661939    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="agent"
Sep 17 21:04:27.662087 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.661945    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="agent"
Sep 17 21:04:27.662087 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.661951    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f2467cf-97ac-40f4-8c8c-3468b677908c" containerName="kserve-container"
Sep 17 21:04:27.662087 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.661956    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f2467cf-97ac-40f4-8c8c-3468b677908c" containerName="kserve-container"
Sep 17 21:04:27.662087 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.661963    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ffdcb55-f68a-4cbf-941a-0a6c817afc12" containerName="storage-initializer"
Sep 17 21:04:27.662087 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.661970    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ffdcb55-f68a-4cbf-941a-0a6c817afc12" containerName="storage-initializer"
Sep 17 21:04:27.662087 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.661976    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ffdcb55-f68a-4cbf-941a-0a6c817afc12" containerName="kserve-container"
Sep 17 21:04:27.662087 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.661982    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ffdcb55-f68a-4cbf-941a-0a6c817afc12" containerName="kserve-container"
Sep 17 21:04:27.662087 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.661988    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7d49a27-d465-49af-8f11-11d586d53a21" containerName="kserve-container"
Sep 17 21:04:27.662087 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.661993    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7d49a27-d465-49af-8f11-11d586d53a21" containerName="kserve-container"
Sep 17 21:04:27.662087 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.661999    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="storage-initializer"
Sep 17 21:04:27.662087 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.662004    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="storage-initializer"
Sep 17 21:04:27.662087 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.662012    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="kube-rbac-proxy"
Sep 17 21:04:27.662087 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.662016    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="kube-rbac-proxy"
Sep 17 21:04:27.662087 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.662022    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ffdcb55-f68a-4cbf-941a-0a6c817afc12" containerName="kube-rbac-proxy"
Sep 17 21:04:27.662087 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.662026    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ffdcb55-f68a-4cbf-941a-0a6c817afc12" containerName="kube-rbac-proxy"
Sep 17 21:04:27.662087 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.662066    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f2467cf-97ac-40f4-8c8c-3468b677908c" containerName="kube-rbac-proxy"
Sep 17 21:04:27.662087 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.662074    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ffdcb55-f68a-4cbf-941a-0a6c817afc12" containerName="kube-rbac-proxy"
Sep 17 21:04:27.662087 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.662080    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f2467cf-97ac-40f4-8c8c-3468b677908c" containerName="kserve-container"
Sep 17 21:04:27.662087 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.662086    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="kube-rbac-proxy"
Sep 17 21:04:27.662087 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.662092    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7d49a27-d465-49af-8f11-11d586d53a21" containerName="kube-rbac-proxy"
Sep 17 21:04:27.662087 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.662097    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7d49a27-d465-49af-8f11-11d586d53a21" containerName="kserve-container"
Sep 17 21:04:27.662816 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.662103    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="agent"
Sep 17 21:04:27.662816 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.662109    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ffdcb55-f68a-4cbf-941a-0a6c817afc12" containerName="kserve-container"
Sep 17 21:04:27.662816 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.662114    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a308e40-a9f1-4a70-aae9-eca1b7b30d37" containerName="kserve-container"
Sep 17 21:04:27.665003 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.664984    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd"
Sep 17 21:04:27.669352 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.669334    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 17 21:04:27.669460 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.669373    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 21:04:27.698433 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.698404    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd"]
Sep 17 21:04:27.755856 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.755826    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ffdcb55-f68a-4cbf-941a-0a6c817afc12-kserve-provision-location\") pod \"8ffdcb55-f68a-4cbf-941a-0a6c817afc12\" (UID: \"8ffdcb55-f68a-4cbf-941a-0a6c817afc12\") "
Sep 17 21:04:27.756008 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.755868    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ffdcb55-f68a-4cbf-941a-0a6c817afc12-proxy-tls\") pod \"8ffdcb55-f68a-4cbf-941a-0a6c817afc12\" (UID: \"8ffdcb55-f68a-4cbf-941a-0a6c817afc12\") "
Sep 17 21:04:27.756008 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.755888    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ffdcb55-f68a-4cbf-941a-0a6c817afc12-kube-rbac-proxy-sar-config\") pod \"8ffdcb55-f68a-4cbf-941a-0a6c817afc12\" (UID: \"8ffdcb55-f68a-4cbf-941a-0a6c817afc12\") "
Sep 17 21:04:27.756008 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.755934    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wvpb\" (UniqueName: \"kubernetes.io/projected/8ffdcb55-f68a-4cbf-941a-0a6c817afc12-kube-api-access-2wvpb\") pod \"8ffdcb55-f68a-4cbf-941a-0a6c817afc12\" (UID: \"8ffdcb55-f68a-4cbf-941a-0a6c817afc12\") "
Sep 17 21:04:27.756008 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.755954    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ffdcb55-f68a-4cbf-941a-0a6c817afc12-cabundle-cert\") pod \"8ffdcb55-f68a-4cbf-941a-0a6c817afc12\" (UID: \"8ffdcb55-f68a-4cbf-941a-0a6c817afc12\") "
Sep 17 21:04:27.756248 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.756114    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6qdl\" (UniqueName: \"kubernetes.io/projected/08b521ce-377f-4f24-ba4a-95585aca48d5-kube-api-access-j6qdl\") pod \"isvc-canary-stop-v2-predictor-d65d59944-ww6pd\" (UID: \"08b521ce-377f-4f24-ba4a-95585aca48d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd"
Sep 17 21:04:27.756248 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.756192    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08b521ce-377f-4f24-ba4a-95585aca48d5-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-d65d59944-ww6pd\" (UID: \"08b521ce-377f-4f24-ba4a-95585aca48d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd"
Sep 17 21:04:27.756336 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.756257    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ffdcb55-f68a-4cbf-941a-0a6c817afc12-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8ffdcb55-f68a-4cbf-941a-0a6c817afc12" (UID: "8ffdcb55-f68a-4cbf-941a-0a6c817afc12"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:04:27.756336 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.756313    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08b521ce-377f-4f24-ba4a-95585aca48d5-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-d65d59944-ww6pd\" (UID: \"08b521ce-377f-4f24-ba4a-95585aca48d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd"
Sep 17 21:04:27.756410 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.756338    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ffdcb55-f68a-4cbf-941a-0a6c817afc12-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8ffdcb55-f68a-4cbf-941a-0a6c817afc12" (UID: "8ffdcb55-f68a-4cbf-941a-0a6c817afc12"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:04:27.756410 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.756347    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ffdcb55-f68a-4cbf-941a-0a6c817afc12-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8ffdcb55-f68a-4cbf-941a-0a6c817afc12" (UID: "8ffdcb55-f68a-4cbf-941a-0a6c817afc12"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:04:27.756410 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.756384    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08b521ce-377f-4f24-ba4a-95585aca48d5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-d65d59944-ww6pd\" (UID: \"08b521ce-377f-4f24-ba4a-95585aca48d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd"
Sep 17 21:04:27.756550 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.756426    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08b521ce-377f-4f24-ba4a-95585aca48d5-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-d65d59944-ww6pd\" (UID: \"08b521ce-377f-4f24-ba4a-95585aca48d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd"
Sep 17 21:04:27.756550 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.756480    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ffdcb55-f68a-4cbf-941a-0a6c817afc12-cabundle-cert\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:04:27.756550 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.756492    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ffdcb55-f68a-4cbf-941a-0a6c817afc12-kserve-provision-location\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:04:27.756550 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.756500    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ffdcb55-f68a-4cbf-941a-0a6c817afc12-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:04:27.757881 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.757863    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ffdcb55-f68a-4cbf-941a-0a6c817afc12-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8ffdcb55-f68a-4cbf-941a-0a6c817afc12" (UID: "8ffdcb55-f68a-4cbf-941a-0a6c817afc12"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:04:27.758123 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.758101    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ffdcb55-f68a-4cbf-941a-0a6c817afc12-kube-api-access-2wvpb" (OuterVolumeSpecName: "kube-api-access-2wvpb") pod "8ffdcb55-f68a-4cbf-941a-0a6c817afc12" (UID: "8ffdcb55-f68a-4cbf-941a-0a6c817afc12"). InnerVolumeSpecName "kube-api-access-2wvpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:04:27.857243 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.857190    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08b521ce-377f-4f24-ba4a-95585aca48d5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-d65d59944-ww6pd\" (UID: \"08b521ce-377f-4f24-ba4a-95585aca48d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd"
Sep 17 21:04:27.857403 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.857265    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08b521ce-377f-4f24-ba4a-95585aca48d5-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-d65d59944-ww6pd\" (UID: \"08b521ce-377f-4f24-ba4a-95585aca48d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd"
Sep 17 21:04:27.857403 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.857304    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6qdl\" (UniqueName: \"kubernetes.io/projected/08b521ce-377f-4f24-ba4a-95585aca48d5-kube-api-access-j6qdl\") pod \"isvc-canary-stop-v2-predictor-d65d59944-ww6pd\" (UID: \"08b521ce-377f-4f24-ba4a-95585aca48d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd"
Sep 17 21:04:27.857403 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.857358    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08b521ce-377f-4f24-ba4a-95585aca48d5-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-d65d59944-ww6pd\" (UID: \"08b521ce-377f-4f24-ba4a-95585aca48d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd"
Sep 17 21:04:27.857403 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.857387    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08b521ce-377f-4f24-ba4a-95585aca48d5-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-d65d59944-ww6pd\" (UID: \"08b521ce-377f-4f24-ba4a-95585aca48d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd"
Sep 17 21:04:27.857626 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.857435    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wvpb\" (UniqueName: \"kubernetes.io/projected/8ffdcb55-f68a-4cbf-941a-0a6c817afc12-kube-api-access-2wvpb\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:04:27.857626 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.857454    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ffdcb55-f68a-4cbf-941a-0a6c817afc12-proxy-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:04:27.857626 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:04:27.857541    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 17 21:04:27.857626 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:04:27.857609    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08b521ce-377f-4f24-ba4a-95585aca48d5-proxy-tls podName:08b521ce-377f-4f24-ba4a-95585aca48d5 nodeName:}" failed. No retries permitted until 2026-09-17 21:04:28.357587231 +0000 UTC m=+909.968320440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/08b521ce-377f-4f24-ba4a-95585aca48d5-proxy-tls") pod "isvc-canary-stop-v2-predictor-d65d59944-ww6pd" (UID: "08b521ce-377f-4f24-ba4a-95585aca48d5") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 17 21:04:27.857778 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.857753    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08b521ce-377f-4f24-ba4a-95585aca48d5-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-d65d59944-ww6pd\" (UID: \"08b521ce-377f-4f24-ba4a-95585aca48d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd"
Sep 17 21:04:27.857932 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.857914    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08b521ce-377f-4f24-ba4a-95585aca48d5-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-d65d59944-ww6pd\" (UID: \"08b521ce-377f-4f24-ba4a-95585aca48d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd"
Sep 17 21:04:27.857973 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.857935    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08b521ce-377f-4f24-ba4a-95585aca48d5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-d65d59944-ww6pd\" (UID: \"08b521ce-377f-4f24-ba4a-95585aca48d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd"
Sep 17 21:04:27.876947 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.876925    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6qdl\" (UniqueName: \"kubernetes.io/projected/08b521ce-377f-4f24-ba4a-95585aca48d5-kube-api-access-j6qdl\") pod \"isvc-canary-stop-v2-predictor-d65d59944-ww6pd\" (UID: \"08b521ce-377f-4f24-ba4a-95585aca48d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd"
Sep 17 21:04:27.877118 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.877101    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"]
Sep 17 21:04:27.881738 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.881722    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"
Sep 17 21:04:27.887856 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:04:27.887831    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"isvc-canary-stop-predictor-55656fdf77-hm692\" is forbidden: User \"system:node:ip-10-0-138-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-138-140.ec2.internal' and this object" podUID="77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"
Sep 17 21:04:27.888391 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:04:27.888369    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"isvc-canary-stop-predictor-serving-cert\" is forbidden: User \"system:node:ip-10-0-138-140.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-138-140.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\"" type="*v1.Secret"
Sep 17 21:04:27.907906 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.907882    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"]
Sep 17 21:04:27.958739 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.958676    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-55656fdf77-hm692\" (UID: \"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"
Sep 17 21:04:27.958860 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.958736    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-proxy-tls\") pod \"isvc-canary-stop-predictor-55656fdf77-hm692\" (UID: \"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"
Sep 17 21:04:27.958860 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.958771    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wlb5\" (UniqueName: \"kubernetes.io/projected/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-kube-api-access-2wlb5\") pod \"isvc-canary-stop-predictor-55656fdf77-hm692\" (UID: \"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"
Sep 17 21:04:27.958860 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.958793    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-cabundle-cert\") pod \"isvc-canary-stop-predictor-55656fdf77-hm692\" (UID: \"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"
Sep 17 21:04:27.958860 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:27.958814    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-kserve-provision-location\") pod \"isvc-canary-stop-predictor-55656fdf77-hm692\" (UID: \"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"
Sep 17 21:04:28.060091 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:28.060063    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-kserve-provision-location\") pod \"isvc-canary-stop-predictor-55656fdf77-hm692\" (UID: \"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"
Sep 17 21:04:28.060230 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:28.060119    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-55656fdf77-hm692\" (UID: \"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"
Sep 17 21:04:28.060230 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:28.060170    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-proxy-tls\") pod \"isvc-canary-stop-predictor-55656fdf77-hm692\" (UID: \"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"
Sep 17 21:04:28.060230 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:28.060203    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wlb5\" (UniqueName: \"kubernetes.io/projected/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-kube-api-access-2wlb5\") pod \"isvc-canary-stop-predictor-55656fdf77-hm692\" (UID: \"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"
Sep 17 21:04:28.060379 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:28.060257    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-cabundle-cert\") pod \"isvc-canary-stop-predictor-55656fdf77-hm692\" (UID: \"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"
Sep 17 21:04:28.060537 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:28.060516    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-kserve-provision-location\") pod \"isvc-canary-stop-predictor-55656fdf77-hm692\" (UID: \"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"
Sep 17 21:04:28.060830 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:28.060813    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-55656fdf77-hm692\" (UID: \"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"
Sep 17 21:04:28.060868 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:28.060818    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-cabundle-cert\") pod \"isvc-canary-stop-predictor-55656fdf77-hm692\" (UID: \"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"
Sep 17 21:04:28.070097 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:28.070074    2566 generic.go:356] "Generic (PLEG): container finished" podID="8ffdcb55-f68a-4cbf-941a-0a6c817afc12" containerID="64bc584f3eab94ac0d3b36cab01728b8aa6a8bd8402f2acaf4e2e8226e9806ae" exitCode=0
Sep 17 21:04:28.070187 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:28.070139    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv"
Sep 17 21:04:28.070249 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:28.070141    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv" event={"ID":"8ffdcb55-f68a-4cbf-941a-0a6c817afc12","Type":"ContainerDied","Data":"64bc584f3eab94ac0d3b36cab01728b8aa6a8bd8402f2acaf4e2e8226e9806ae"}
Sep 17 21:04:28.070284 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:28.070271    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv" event={"ID":"8ffdcb55-f68a-4cbf-941a-0a6c817afc12","Type":"ContainerDied","Data":"f46179a5937b57f378ab0b0d9dea35817c3335016b8aaec0fb82751f615132b4"}
Sep 17 21:04:28.070313 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:28.070300    2566 scope.go:117] "RemoveContainer" containerID="c920df1e6eef61e2b658c7ba887c96e92d478b874c90b6e4bdfddc215ffb4222"
Sep 17 21:04:28.074961 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:28.074943    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wlb5\" (UniqueName: \"kubernetes.io/projected/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-kube-api-access-2wlb5\") pod \"isvc-canary-stop-predictor-55656fdf77-hm692\" (UID: \"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"
Sep 17 21:04:28.078297 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:28.078279    2566 scope.go:117] "RemoveContainer" containerID="64bc584f3eab94ac0d3b36cab01728b8aa6a8bd8402f2acaf4e2e8226e9806ae"
Sep 17 21:04:28.085058 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:28.085042    2566 scope.go:117] "RemoveContainer" containerID="5683f6cab799adb79d2a0ff9cae065c19f0c741aeffb1d26320b77b588e04182"
Sep 17 21:04:28.091841 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:28.091823    2566 scope.go:117] "RemoveContainer" containerID="c920df1e6eef61e2b658c7ba887c96e92d478b874c90b6e4bdfddc215ffb4222"
Sep 17 21:04:28.092102 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:04:28.092085    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c920df1e6eef61e2b658c7ba887c96e92d478b874c90b6e4bdfddc215ffb4222\": container with ID starting with c920df1e6eef61e2b658c7ba887c96e92d478b874c90b6e4bdfddc215ffb4222 not found: ID does not exist" containerID="c920df1e6eef61e2b658c7ba887c96e92d478b874c90b6e4bdfddc215ffb4222"
Sep 17 21:04:28.092153 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:28.092110    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c920df1e6eef61e2b658c7ba887c96e92d478b874c90b6e4bdfddc215ffb4222"} err="failed to get container status \"c920df1e6eef61e2b658c7ba887c96e92d478b874c90b6e4bdfddc215ffb4222\": rpc error: code = NotFound desc = could not find container \"c920df1e6eef61e2b658c7ba887c96e92d478b874c90b6e4bdfddc215ffb4222\": container with ID starting with c920df1e6eef61e2b658c7ba887c96e92d478b874c90b6e4bdfddc215ffb4222 not found: ID does not exist"
Sep 17 21:04:28.092153 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:28.092126    2566 scope.go:117] "RemoveContainer" containerID="64bc584f3eab94ac0d3b36cab01728b8aa6a8bd8402f2acaf4e2e8226e9806ae"
Sep 17 21:04:28.092376 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:04:28.092362    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64bc584f3eab94ac0d3b36cab01728b8aa6a8bd8402f2acaf4e2e8226e9806ae\": container with ID starting with 64bc584f3eab94ac0d3b36cab01728b8aa6a8bd8402f2acaf4e2e8226e9806ae not found: ID does not exist" containerID="64bc584f3eab94ac0d3b36cab01728b8aa6a8bd8402f2acaf4e2e8226e9806ae"
Sep 17 21:04:28.092417 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:28.092379    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64bc584f3eab94ac0d3b36cab01728b8aa6a8bd8402f2acaf4e2e8226e9806ae"} err="failed to get container status \"64bc584f3eab94ac0d3b36cab01728b8aa6a8bd8402f2acaf4e2e8226e9806ae\": rpc error: code = NotFound desc = could not find container \"64bc584f3eab94ac0d3b36cab01728b8aa6a8bd8402f2acaf4e2e8226e9806ae\": container with ID starting with 64bc584f3eab94ac0d3b36cab01728b8aa6a8bd8402f2acaf4e2e8226e9806ae not found: ID does not exist"
Sep 17 21:04:28.092417 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:28.092392    2566 scope.go:117] "RemoveContainer" containerID="5683f6cab799adb79d2a0ff9cae065c19f0c741aeffb1d26320b77b588e04182"
Sep 17 21:04:28.092664 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:04:28.092648    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5683f6cab799adb79d2a0ff9cae065c19f0c741aeffb1d26320b77b588e04182\": container with ID starting with 5683f6cab799adb79d2a0ff9cae065c19f0c741aeffb1d26320b77b588e04182 not found: ID does not exist" containerID="5683f6cab799adb79d2a0ff9cae065c19f0c741aeffb1d26320b77b588e04182"
Sep 17 21:04:28.092714 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:28.092669    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5683f6cab799adb79d2a0ff9cae065c19f0c741aeffb1d26320b77b588e04182"} err="failed to get container status \"5683f6cab799adb79d2a0ff9cae065c19f0c741aeffb1d26320b77b588e04182\": rpc error: code = NotFound desc = could not find container \"5683f6cab799adb79d2a0ff9cae065c19f0c741aeffb1d26320b77b588e04182\": container with ID starting with 5683f6cab799adb79d2a0ff9cae065c19f0c741aeffb1d26320b77b588e04182 not found: ID does not exist"
Sep 17 21:04:28.162067 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:28.162039    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv"]
Sep 17 21:04:28.173422 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:28.173398    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lfppv"]
Sep 17 21:04:28.362260 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:28.362236    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08b521ce-377f-4f24-ba4a-95585aca48d5-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-d65d59944-ww6pd\" (UID: \"08b521ce-377f-4f24-ba4a-95585aca48d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd"
Sep 17 21:04:28.364702 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:28.364682    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08b521ce-377f-4f24-ba4a-95585aca48d5-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-d65d59944-ww6pd\" (UID: \"08b521ce-377f-4f24-ba4a-95585aca48d5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd"
Sep 17 21:04:28.459433 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:28.459402    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ffdcb55-f68a-4cbf-941a-0a6c817afc12" path="/var/lib/kubelet/pods/8ffdcb55-f68a-4cbf-941a-0a6c817afc12/volumes"
Sep 17 21:04:28.574631 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:28.574597    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd"
Sep 17 21:04:28.873432 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:28.873410    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd"]
Sep 17 21:04:28.875590 ip-10-0-138-140 kubenswrapper[2566]: W0917 21:04:28.875563    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b521ce_377f_4f24_ba4a_95585aca48d5.slice/crio-77aa688736eb3054abee4d67f409c04a666ed8186fd467488cdd789273d45066 WatchSource:0}: Error finding container 77aa688736eb3054abee4d67f409c04a666ed8186fd467488cdd789273d45066: Status 404 returned error can't find the container with id 77aa688736eb3054abee4d67f409c04a666ed8186fd467488cdd789273d45066
Sep 17 21:04:29.060700 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:04:29.060670    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: failed to sync secret cache: timed out waiting for the condition
Sep 17 21:04:29.060857 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:04:29.060747    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-proxy-tls podName:77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9 nodeName:}" failed. No retries permitted until 2026-09-17 21:04:29.560728092 +0000 UTC m=+911.171461300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-proxy-tls") pod "isvc-canary-stop-predictor-55656fdf77-hm692" (UID: "77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9") : failed to sync secret cache: timed out waiting for the condition
Sep 17 21:04:29.077578 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:29.077546    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd" event={"ID":"08b521ce-377f-4f24-ba4a-95585aca48d5","Type":"ContainerStarted","Data":"5573708bc3b434048d5b30be7e7c7ba686428affb32f99f1647436f5097cf302"}
Sep 17 21:04:29.077905 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:29.077608    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd" event={"ID":"08b521ce-377f-4f24-ba4a-95585aca48d5","Type":"ContainerStarted","Data":"77aa688736eb3054abee4d67f409c04a666ed8186fd467488cdd789273d45066"}
Sep 17 21:04:29.278392 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:29.278360    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 21:04:29.572995 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:29.572907    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-proxy-tls\") pod \"isvc-canary-stop-predictor-55656fdf77-hm692\" (UID: \"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"
Sep 17 21:04:29.575310 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:29.575286    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-proxy-tls\") pod \"isvc-canary-stop-predictor-55656fdf77-hm692\" (UID: \"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"
Sep 17 21:04:29.691115 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:29.691087    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"
Sep 17 21:04:29.963015 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:29.962992    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"]
Sep 17 21:04:29.964771 ip-10-0-138-140 kubenswrapper[2566]: W0917 21:04:29.964745    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77dcbaf7_6ebd_45c9_87c0_c1fb3ab31cc9.slice/crio-65e2c3d2b76715ad0100f25aab344da3347c0bf63e9414d6f71bf9186ace4edc WatchSource:0}: Error finding container 65e2c3d2b76715ad0100f25aab344da3347c0bf63e9414d6f71bf9186ace4edc: Status 404 returned error can't find the container with id 65e2c3d2b76715ad0100f25aab344da3347c0bf63e9414d6f71bf9186ace4edc
Sep 17 21:04:30.083229 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:30.083187    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692" event={"ID":"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9","Type":"ContainerStarted","Data":"32598524d3fdd6e2788b0f7fa2d019f6b77dfcde22e31a2ee5a0ad412fca641e"}
Sep 17 21:04:30.083410 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:30.083273    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692" event={"ID":"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9","Type":"ContainerStarted","Data":"65e2c3d2b76715ad0100f25aab344da3347c0bf63e9414d6f71bf9186ace4edc"}
Sep 17 21:04:33.095969 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:33.095931    2566 generic.go:356] "Generic (PLEG): container finished" podID="08b521ce-377f-4f24-ba4a-95585aca48d5" containerID="5573708bc3b434048d5b30be7e7c7ba686428affb32f99f1647436f5097cf302" exitCode=0
Sep 17 21:04:33.095969 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:33.095970    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd" event={"ID":"08b521ce-377f-4f24-ba4a-95585aca48d5","Type":"ContainerDied","Data":"5573708bc3b434048d5b30be7e7c7ba686428affb32f99f1647436f5097cf302"}
Sep 17 21:04:34.101964 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:34.101930    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd" event={"ID":"08b521ce-377f-4f24-ba4a-95585aca48d5","Type":"ContainerStarted","Data":"928aef75df7a2c1ba045a167fdf41e57f72816756d005836261c911b9fe18cde"}
Sep 17 21:04:34.102385 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:34.102001    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd" event={"ID":"08b521ce-377f-4f24-ba4a-95585aca48d5","Type":"ContainerStarted","Data":"7cd87444496f230ebceb924414eea393b2b8793d3fe165816d2d8a1b06628a1e"}
Sep 17 21:04:34.102385 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:34.102283    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd"
Sep 17 21:04:34.102385 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:34.102378    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd"
Sep 17 21:04:34.103588 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:34.103559    2566 generic.go:356] "Generic (PLEG): container finished" podID="77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" containerID="32598524d3fdd6e2788b0f7fa2d019f6b77dfcde22e31a2ee5a0ad412fca641e" exitCode=0
Sep 17 21:04:34.103693 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:34.103593    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692" event={"ID":"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9","Type":"ContainerDied","Data":"32598524d3fdd6e2788b0f7fa2d019f6b77dfcde22e31a2ee5a0ad412fca641e"}
Sep 17 21:04:34.103877 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:34.103848    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd" podUID="08b521ce-377f-4f24-ba4a-95585aca48d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 21:04:34.129413 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:34.129375    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd" podStartSLOduration=3.445254532 podStartE2EDuration="7.129363205s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.684108673s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:04:34.128982044 +0000 UTC m=+915.739715270" watchObservedRunningTime="2026-09-17 21:04:34.129363205 +0000 UTC m=+915.740096433"
Sep 17 21:04:35.109563 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:35.109525    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692" event={"ID":"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9","Type":"ContainerStarted","Data":"f1f58b99e601386385b772c6dabcde54367d01748efbcd8f98fcc40ef5edae58"}
Sep 17 21:04:35.110055 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:35.109600    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692" event={"ID":"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9","Type":"ContainerStarted","Data":"c2a6c304167b8c6968fe073ded30d533edc6d049a90f4a16522fb079729946e7"}
Sep 17 21:04:35.110055 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:35.109773    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd" podUID="08b521ce-377f-4f24-ba4a-95585aca48d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 21:04:35.110055 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:35.109930    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"
Sep 17 21:04:35.110055 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:35.110052    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"
Sep 17 21:04:35.111167 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:35.111141    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692" podUID="77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 21:04:35.139221 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:35.139169    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692" podStartSLOduration=4.253428305 podStartE2EDuration="8.139152552s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.885724247s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:04:35.138641353 +0000 UTC m=+916.749374584" watchObservedRunningTime="2026-09-17 21:04:35.139152552 +0000 UTC m=+916.749885783"
Sep 17 21:04:36.113993 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:36.113956    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692" podUID="77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 21:04:40.114330 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:40.114303    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd"
Sep 17 21:04:40.116679 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:40.114877    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd" podUID="08b521ce-377f-4f24-ba4a-95585aca48d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 21:04:41.119351 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:41.119321    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"
Sep 17 21:04:41.120257 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:41.120226    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692" podUID="77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 21:04:50.115736 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:50.115696    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd" podUID="08b521ce-377f-4f24-ba4a-95585aca48d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 21:04:51.120570 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:04:51.120531    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692" podUID="77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 21:05:00.114923 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:00.114885    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd" podUID="08b521ce-377f-4f24-ba4a-95585aca48d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 21:05:01.121120 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:01.121078    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692" podUID="77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 21:05:03.231007 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.230969    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm"]
Sep 17 21:05:03.234781 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.234760    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm"
Sep 17 21:05:03.237324 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.237306    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-e3e0d-kube-rbac-proxy-sar-config\""
Sep 17 21:05:03.237463 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.237447    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-e3e0d-predictor-serving-cert\""
Sep 17 21:05:03.248579 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.248555    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm"]
Sep 17 21:05:03.318991 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.318958    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5"]
Sep 17 21:05:03.322367 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.322349    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5"
Sep 17 21:05:03.325149 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.325136    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-e3e0d-kube-rbac-proxy-sar-config\""
Sep 17 21:05:03.325228 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.325138    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-e3e0d-predictor-serving-cert\""
Sep 17 21:05:03.331734 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.331710    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtpkh\" (UniqueName: \"kubernetes.io/projected/199305ac-35b9-4ccc-9ef4-7bb783c42118-kube-api-access-wtpkh\") pod \"isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm\" (UID: \"199305ac-35b9-4ccc-9ef4-7bb783c42118\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm"
Sep 17 21:05:03.331820 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.331741    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/199305ac-35b9-4ccc-9ef4-7bb783c42118-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm\" (UID: \"199305ac-35b9-4ccc-9ef4-7bb783c42118\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm"
Sep 17 21:05:03.331820 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.331805    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/199305ac-35b9-4ccc-9ef4-7bb783c42118-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm\" (UID: \"199305ac-35b9-4ccc-9ef4-7bb783c42118\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm"
Sep 17 21:05:03.331898 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.331833    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/199305ac-35b9-4ccc-9ef4-7bb783c42118-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm\" (UID: \"199305ac-35b9-4ccc-9ef4-7bb783c42118\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm"
Sep 17 21:05:03.331898 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.331865    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/199305ac-35b9-4ccc-9ef4-7bb783c42118-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm\" (UID: \"199305ac-35b9-4ccc-9ef4-7bb783c42118\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm"
Sep 17 21:05:03.334723 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.334701    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5"]
Sep 17 21:05:03.432931 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.432891    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/199305ac-35b9-4ccc-9ef4-7bb783c42118-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm\" (UID: \"199305ac-35b9-4ccc-9ef4-7bb783c42118\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm"
Sep 17 21:05:03.432931 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.432953    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/199305ac-35b9-4ccc-9ef4-7bb783c42118-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm\" (UID: \"199305ac-35b9-4ccc-9ef4-7bb783c42118\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm"
Sep 17 21:05:03.433198 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.433078    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/432c0a3e-23a6-45ba-81f4-7cae8de3f677-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5\" (UID: \"432c0a3e-23a6-45ba-81f4-7cae8de3f677\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5"
Sep 17 21:05:03.433198 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.433125    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/199305ac-35b9-4ccc-9ef4-7bb783c42118-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm\" (UID: \"199305ac-35b9-4ccc-9ef4-7bb783c42118\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm"
Sep 17 21:05:03.433198 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.433172    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtpkh\" (UniqueName: \"kubernetes.io/projected/199305ac-35b9-4ccc-9ef4-7bb783c42118-kube-api-access-wtpkh\") pod \"isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm\" (UID: \"199305ac-35b9-4ccc-9ef4-7bb783c42118\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm"
Sep 17 21:05:03.433198 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.433193    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/432c0a3e-23a6-45ba-81f4-7cae8de3f677-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5\" (UID: \"432c0a3e-23a6-45ba-81f4-7cae8de3f677\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5"
Sep 17 21:05:03.433402 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.433243    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/199305ac-35b9-4ccc-9ef4-7bb783c42118-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm\" (UID: \"199305ac-35b9-4ccc-9ef4-7bb783c42118\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm"
Sep 17 21:05:03.433402 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.433270    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/432c0a3e-23a6-45ba-81f4-7cae8de3f677-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5\" (UID: \"432c0a3e-23a6-45ba-81f4-7cae8de3f677\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5"
Sep 17 21:05:03.433402 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.433308    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5q7k\" (UniqueName: \"kubernetes.io/projected/432c0a3e-23a6-45ba-81f4-7cae8de3f677-kube-api-access-j5q7k\") pod \"isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5\" (UID: \"432c0a3e-23a6-45ba-81f4-7cae8de3f677\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5"
Sep 17 21:05:03.433402 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.433344    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/432c0a3e-23a6-45ba-81f4-7cae8de3f677-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5\" (UID: \"432c0a3e-23a6-45ba-81f4-7cae8de3f677\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5"
Sep 17 21:05:03.433560 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.433490    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/199305ac-35b9-4ccc-9ef4-7bb783c42118-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm\" (UID: \"199305ac-35b9-4ccc-9ef4-7bb783c42118\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm"
Sep 17 21:05:03.433679 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.433652    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/199305ac-35b9-4ccc-9ef4-7bb783c42118-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm\" (UID: \"199305ac-35b9-4ccc-9ef4-7bb783c42118\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm"
Sep 17 21:05:03.433679 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.433678    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/199305ac-35b9-4ccc-9ef4-7bb783c42118-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm\" (UID: \"199305ac-35b9-4ccc-9ef4-7bb783c42118\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm"
Sep 17 21:05:03.435567 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.435546    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/199305ac-35b9-4ccc-9ef4-7bb783c42118-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm\" (UID: \"199305ac-35b9-4ccc-9ef4-7bb783c42118\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm"
Sep 17 21:05:03.442908 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.442890    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtpkh\" (UniqueName: \"kubernetes.io/projected/199305ac-35b9-4ccc-9ef4-7bb783c42118-kube-api-access-wtpkh\") pod \"isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm\" (UID: \"199305ac-35b9-4ccc-9ef4-7bb783c42118\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm"
Sep 17 21:05:03.534232 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.534147    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/432c0a3e-23a6-45ba-81f4-7cae8de3f677-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5\" (UID: \"432c0a3e-23a6-45ba-81f4-7cae8de3f677\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5"
Sep 17 21:05:03.534232 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.534185    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5q7k\" (UniqueName: \"kubernetes.io/projected/432c0a3e-23a6-45ba-81f4-7cae8de3f677-kube-api-access-j5q7k\") pod \"isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5\" (UID: \"432c0a3e-23a6-45ba-81f4-7cae8de3f677\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5"
Sep 17 21:05:03.534232 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.534223    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/432c0a3e-23a6-45ba-81f4-7cae8de3f677-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5\" (UID: \"432c0a3e-23a6-45ba-81f4-7cae8de3f677\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5"
Sep 17 21:05:03.534444 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:05:03.534317    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-e3e0d-predictor-serving-cert" not found
Sep 17 21:05:03.534444 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:05:03.534373    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/432c0a3e-23a6-45ba-81f4-7cae8de3f677-proxy-tls podName:432c0a3e-23a6-45ba-81f4-7cae8de3f677 nodeName:}" failed. No retries permitted until 2026-09-17 21:05:04.034357312 +0000 UTC m=+945.645090519 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/432c0a3e-23a6-45ba-81f4-7cae8de3f677-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5" (UID: "432c0a3e-23a6-45ba-81f4-7cae8de3f677") : secret "isvc-xgboost-graph-raw-hpa-e3e0d-predictor-serving-cert" not found
Sep 17 21:05:03.534444 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.534389    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/432c0a3e-23a6-45ba-81f4-7cae8de3f677-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5\" (UID: \"432c0a3e-23a6-45ba-81f4-7cae8de3f677\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5"
Sep 17 21:05:03.534571 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.534442    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/432c0a3e-23a6-45ba-81f4-7cae8de3f677-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5\" (UID: \"432c0a3e-23a6-45ba-81f4-7cae8de3f677\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5"
Sep 17 21:05:03.534757 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.534737    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/432c0a3e-23a6-45ba-81f4-7cae8de3f677-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5\" (UID: \"432c0a3e-23a6-45ba-81f4-7cae8de3f677\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5"
Sep 17 21:05:03.534820 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.534802    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/432c0a3e-23a6-45ba-81f4-7cae8de3f677-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5\" (UID: \"432c0a3e-23a6-45ba-81f4-7cae8de3f677\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5"
Sep 17 21:05:03.534940 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.534922    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/432c0a3e-23a6-45ba-81f4-7cae8de3f677-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5\" (UID: \"432c0a3e-23a6-45ba-81f4-7cae8de3f677\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5"
Sep 17 21:05:03.545005 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.544979    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm"
Sep 17 21:05:03.545800 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.545783    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5q7k\" (UniqueName: \"kubernetes.io/projected/432c0a3e-23a6-45ba-81f4-7cae8de3f677-kube-api-access-j5q7k\") pod \"isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5\" (UID: \"432c0a3e-23a6-45ba-81f4-7cae8de3f677\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5"
Sep 17 21:05:03.818236 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:03.818009    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm"]
Sep 17 21:05:03.820349 ip-10-0-138-140 kubenswrapper[2566]: W0917 21:05:03.820325    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod199305ac_35b9_4ccc_9ef4_7bb783c42118.slice/crio-e062930eccf1518f78b843699a5968d271e3a65ff2730abd1397f124fecfbdd4 WatchSource:0}: Error finding container e062930eccf1518f78b843699a5968d271e3a65ff2730abd1397f124fecfbdd4: Status 404 returned error can't find the container with id e062930eccf1518f78b843699a5968d271e3a65ff2730abd1397f124fecfbdd4
Sep 17 21:05:04.039067 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:04.039027    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/432c0a3e-23a6-45ba-81f4-7cae8de3f677-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5\" (UID: \"432c0a3e-23a6-45ba-81f4-7cae8de3f677\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5"
Sep 17 21:05:04.041315 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:04.041289    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/432c0a3e-23a6-45ba-81f4-7cae8de3f677-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5\" (UID: \"432c0a3e-23a6-45ba-81f4-7cae8de3f677\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5"
Sep 17 21:05:04.225522 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:04.225480    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm" event={"ID":"199305ac-35b9-4ccc-9ef4-7bb783c42118","Type":"ContainerStarted","Data":"79e14833b1ce1f31fb97edd02b4a75ea1ab11165ec2ac2e50799d15333446d25"}
Sep 17 21:05:04.225675 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:04.225557    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm" event={"ID":"199305ac-35b9-4ccc-9ef4-7bb783c42118","Type":"ContainerStarted","Data":"e062930eccf1518f78b843699a5968d271e3a65ff2730abd1397f124fecfbdd4"}
Sep 17 21:05:04.232390 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:04.232371    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5"
Sep 17 21:05:04.503858 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:04.503836    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5"]
Sep 17 21:05:04.506002 ip-10-0-138-140 kubenswrapper[2566]: W0917 21:05:04.505978    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432c0a3e_23a6_45ba_81f4_7cae8de3f677.slice/crio-bded7c4dccfe92025b1a0a80fb48776a23dd2a25b5ad0882f29687d56d3001c9 WatchSource:0}: Error finding container bded7c4dccfe92025b1a0a80fb48776a23dd2a25b5ad0882f29687d56d3001c9: Status 404 returned error can't find the container with id bded7c4dccfe92025b1a0a80fb48776a23dd2a25b5ad0882f29687d56d3001c9
Sep 17 21:05:05.230814 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:05.230776    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5" event={"ID":"432c0a3e-23a6-45ba-81f4-7cae8de3f677","Type":"ContainerStarted","Data":"fd7b5896ffb7b037fb3d3a063cee9b3631cde1a93a3cbb795b3e6760791e2413"}
Sep 17 21:05:05.230999 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:05.230836    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5" event={"ID":"432c0a3e-23a6-45ba-81f4-7cae8de3f677","Type":"ContainerStarted","Data":"bded7c4dccfe92025b1a0a80fb48776a23dd2a25b5ad0882f29687d56d3001c9"}
Sep 17 21:05:08.244437 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:08.244407    2566 generic.go:356] "Generic (PLEG): container finished" podID="432c0a3e-23a6-45ba-81f4-7cae8de3f677" containerID="fd7b5896ffb7b037fb3d3a063cee9b3631cde1a93a3cbb795b3e6760791e2413" exitCode=0
Sep 17 21:05:08.244806 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:08.244477    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5" event={"ID":"432c0a3e-23a6-45ba-81f4-7cae8de3f677","Type":"ContainerDied","Data":"fd7b5896ffb7b037fb3d3a063cee9b3631cde1a93a3cbb795b3e6760791e2413"}
Sep 17 21:05:08.246089 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:08.246068    2566 generic.go:356] "Generic (PLEG): container finished" podID="199305ac-35b9-4ccc-9ef4-7bb783c42118" containerID="79e14833b1ce1f31fb97edd02b4a75ea1ab11165ec2ac2e50799d15333446d25" exitCode=0
Sep 17 21:05:08.246252 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:08.246109    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm" event={"ID":"199305ac-35b9-4ccc-9ef4-7bb783c42118","Type":"ContainerDied","Data":"79e14833b1ce1f31fb97edd02b4a75ea1ab11165ec2ac2e50799d15333446d25"}
Sep 17 21:05:09.254779 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:09.254704    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm" event={"ID":"199305ac-35b9-4ccc-9ef4-7bb783c42118","Type":"ContainerStarted","Data":"7f0d0f2bd79e1332d65db51a22128b5564b5d777030a57badabc445dcd1f01e5"}
Sep 17 21:05:09.255286 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:09.254792    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm" event={"ID":"199305ac-35b9-4ccc-9ef4-7bb783c42118","Type":"ContainerStarted","Data":"e21eb6e3f7526e220b91f8ffc2d375b30a661de17b098eb56f59d2178a9c3182"}
Sep 17 21:05:09.255286 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:09.255015    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm"
Sep 17 21:05:09.276551 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:09.276492    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm" podStartSLOduration=2.772086632 podStartE2EDuration="6.276472954s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.504386322s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:05:09.273478912 +0000 UTC m=+950.884212142" watchObservedRunningTime="2026-09-17 21:05:09.276472954 +0000 UTC m=+950.887206184"
Sep 17 21:05:10.114965 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:10.114911    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd" podUID="08b521ce-377f-4f24-ba4a-95585aca48d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 21:05:10.260902 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:10.260818    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm"
Sep 17 21:05:10.262262 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:10.262204    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm" podUID="199305ac-35b9-4ccc-9ef4-7bb783c42118" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 21:05:11.120484 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:11.120447    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692" podUID="77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 21:05:11.266343 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:11.266302    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm" podUID="199305ac-35b9-4ccc-9ef4-7bb783c42118" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 21:05:16.271577 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:16.271538    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm"
Sep 17 21:05:16.272297 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:16.272260    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm" podUID="199305ac-35b9-4ccc-9ef4-7bb783c42118" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 21:05:20.115117 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:20.115079    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd" podUID="08b521ce-377f-4f24-ba4a-95585aca48d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 21:05:21.120225 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:21.120168    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692" podUID="77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 21:05:26.273040 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:26.273001    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm" podUID="199305ac-35b9-4ccc-9ef4-7bb783c42118" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 21:05:27.352164 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:27.352129    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5" event={"ID":"432c0a3e-23a6-45ba-81f4-7cae8de3f677","Type":"ContainerStarted","Data":"eb133b3d320148436296ce780efd7b1620b897af5519b12e078ddb577073b6e8"}
Sep 17 21:05:27.352576 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:27.352234    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5" event={"ID":"432c0a3e-23a6-45ba-81f4-7cae8de3f677","Type":"ContainerStarted","Data":"0ce45761da2b924c6425c41f781134c97dcaef76b2fc9997111f0c8ac99e5487"}
Sep 17 21:05:27.352576 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:27.352487    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5"
Sep 17 21:05:27.372756 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:27.372712    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5" podStartSLOduration=2.453803104 podStartE2EDuration="24.372697242s" totalImagesPullingTime="18.31046377s" totalInitContainerRuntime="3.608430368s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:05:27.372147735 +0000 UTC m=+968.982880982" watchObservedRunningTime="2026-09-17 21:05:27.372697242 +0000 UTC m=+968.983430473"
Sep 17 21:05:28.356273 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:28.356241    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5"
Sep 17 21:05:28.357317 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:28.357292    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5" podUID="432c0a3e-23a6-45ba-81f4-7cae8de3f677" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 21:05:29.360089 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:29.360048    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5" podUID="432c0a3e-23a6-45ba-81f4-7cae8de3f677" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 21:05:30.115468 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:30.115432    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd" podUID="08b521ce-377f-4f24-ba4a-95585aca48d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 21:05:31.121020 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:31.120982    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692" podUID="77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 21:05:34.365553 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:34.365486    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5"
Sep 17 21:05:34.366059 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:34.366031    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5" podUID="432c0a3e-23a6-45ba-81f4-7cae8de3f677" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 21:05:36.272719 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:36.272676    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm" podUID="199305ac-35b9-4ccc-9ef4-7bb783c42118" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 21:05:40.114793 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:40.114754    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd" podUID="08b521ce-377f-4f24-ba4a-95585aca48d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 21:05:41.121167 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:41.121127    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692" podUID="77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 21:05:44.366779 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:44.366740    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5" podUID="432c0a3e-23a6-45ba-81f4-7cae8de3f677" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 21:05:46.272678 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:46.272640    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm" podUID="199305ac-35b9-4ccc-9ef4-7bb783c42118" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 21:05:50.114899 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:50.114862    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd" podUID="08b521ce-377f-4f24-ba4a-95585aca48d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 21:05:51.120189 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:51.120150    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692" podUID="77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 21:05:54.366053 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:54.366014    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5" podUID="432c0a3e-23a6-45ba-81f4-7cae8de3f677" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 21:05:54.454959 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:54.454922    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd" podUID="08b521ce-377f-4f24-ba4a-95585aca48d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 21:05:56.272205 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:05:56.272166    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm" podUID="199305ac-35b9-4ccc-9ef4-7bb783c42118" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 21:06:01.120376 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:01.120347    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"
Sep 17 21:06:04.366409 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:04.366366    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5" podUID="432c0a3e-23a6-45ba-81f4-7cae8de3f677" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 21:06:04.459688 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:04.459662    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd"
Sep 17 21:06:04.589408 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:04.589373    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"]
Sep 17 21:06:04.589624 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:04.589584    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692" podUID="77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" containerName="kserve-container" containerID="cri-o://c2a6c304167b8c6968fe073ded30d533edc6d049a90f4a16522fb079729946e7" gracePeriod=30
Sep 17 21:06:04.589765 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:04.589621    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692" podUID="77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" containerName="kube-rbac-proxy" containerID="cri-o://f1f58b99e601386385b772c6dabcde54367d01748efbcd8f98fcc40ef5edae58" gracePeriod=30
Sep 17 21:06:05.515268 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:05.515236    2566 generic.go:356] "Generic (PLEG): container finished" podID="77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" containerID="f1f58b99e601386385b772c6dabcde54367d01748efbcd8f98fcc40ef5edae58" exitCode=2
Sep 17 21:06:05.515627 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:05.515308    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692" event={"ID":"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9","Type":"ContainerDied","Data":"f1f58b99e601386385b772c6dabcde54367d01748efbcd8f98fcc40ef5edae58"}
Sep 17 21:06:06.115080 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:06.115041    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692" podUID="77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 17 21:06:06.272666 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:06.272628    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm" podUID="199305ac-35b9-4ccc-9ef4-7bb783c42118" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 21:06:07.547619 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:07.547587    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd"]
Sep 17 21:06:07.548178 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:07.547905    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd" podUID="08b521ce-377f-4f24-ba4a-95585aca48d5" containerName="kserve-container" containerID="cri-o://7cd87444496f230ebceb924414eea393b2b8793d3fe165816d2d8a1b06628a1e" gracePeriod=30
Sep 17 21:06:07.548178 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:07.547977    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd" podUID="08b521ce-377f-4f24-ba4a-95585aca48d5" containerName="kube-rbac-proxy" containerID="cri-o://928aef75df7a2c1ba045a167fdf41e57f72816756d005836261c911b9fe18cde" gracePeriod=30
Sep 17 21:06:08.529477 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:08.529445    2566 generic.go:356] "Generic (PLEG): container finished" podID="08b521ce-377f-4f24-ba4a-95585aca48d5" containerID="928aef75df7a2c1ba045a167fdf41e57f72816756d005836261c911b9fe18cde" exitCode=2
Sep 17 21:06:08.529644 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:08.529516    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd" event={"ID":"08b521ce-377f-4f24-ba4a-95585aca48d5","Type":"ContainerDied","Data":"928aef75df7a2c1ba045a167fdf41e57f72816756d005836261c911b9fe18cde"}
Sep 17 21:06:09.928620 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:09.928598    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"
Sep 17 21:06:10.000037 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.000009    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-kserve-provision-location\") pod \"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9\" (UID: \"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9\") "
Sep 17 21:06:10.000194 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.000071    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-cabundle-cert\") pod \"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9\" (UID: \"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9\") "
Sep 17 21:06:10.000194 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.000110    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-proxy-tls\") pod \"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9\" (UID: \"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9\") "
Sep 17 21:06:10.000194 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.000141    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wlb5\" (UniqueName: \"kubernetes.io/projected/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-kube-api-access-2wlb5\") pod \"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9\" (UID: \"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9\") "
Sep 17 21:06:10.000194 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.000170    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-kube-rbac-proxy-sar-config\") pod \"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9\" (UID: \"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9\") "
Sep 17 21:06:10.000501 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.000361    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" (UID: "77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:06:10.000501 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.000368    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" (UID: "77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:06:10.000598 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.000543    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-kserve-provision-location\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:06:10.000598 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.000557    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-cabundle-cert\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:06:10.000663 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.000611    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" (UID: "77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:06:10.002305 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.002273    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" (UID: "77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:06:10.002401 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.002307    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-kube-api-access-2wlb5" (OuterVolumeSpecName: "kube-api-access-2wlb5") pod "77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" (UID: "77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9"). InnerVolumeSpecName "kube-api-access-2wlb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:06:10.101935 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.101893    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-proxy-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:06:10.101935 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.101932    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wlb5\" (UniqueName: \"kubernetes.io/projected/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-kube-api-access-2wlb5\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:06:10.101935 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.101943    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:06:10.109993 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.109968    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd" podUID="08b521ce-377f-4f24-ba4a-95585aca48d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 17 21:06:10.539190 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.539150    2566 generic.go:356] "Generic (PLEG): container finished" podID="77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" containerID="c2a6c304167b8c6968fe073ded30d533edc6d049a90f4a16522fb079729946e7" exitCode=0
Sep 17 21:06:10.539374 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.539231    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"
Sep 17 21:06:10.539374 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.539249    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692" event={"ID":"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9","Type":"ContainerDied","Data":"c2a6c304167b8c6968fe073ded30d533edc6d049a90f4a16522fb079729946e7"}
Sep 17 21:06:10.539374 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.539317    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692" event={"ID":"77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9","Type":"ContainerDied","Data":"65e2c3d2b76715ad0100f25aab344da3347c0bf63e9414d6f71bf9186ace4edc"}
Sep 17 21:06:10.539374 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.539358    2566 scope.go:117] "RemoveContainer" containerID="f1f58b99e601386385b772c6dabcde54367d01748efbcd8f98fcc40ef5edae58"
Sep 17 21:06:10.547392 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.547379    2566 scope.go:117] "RemoveContainer" containerID="c2a6c304167b8c6968fe073ded30d533edc6d049a90f4a16522fb079729946e7"
Sep 17 21:06:10.554498 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.554463    2566 scope.go:117] "RemoveContainer" containerID="32598524d3fdd6e2788b0f7fa2d019f6b77dfcde22e31a2ee5a0ad412fca641e"
Sep 17 21:06:10.561263 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.561247    2566 scope.go:117] "RemoveContainer" containerID="f1f58b99e601386385b772c6dabcde54367d01748efbcd8f98fcc40ef5edae58"
Sep 17 21:06:10.561550 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:06:10.561532    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1f58b99e601386385b772c6dabcde54367d01748efbcd8f98fcc40ef5edae58\": container with ID starting with f1f58b99e601386385b772c6dabcde54367d01748efbcd8f98fcc40ef5edae58 not found: ID does not exist" containerID="f1f58b99e601386385b772c6dabcde54367d01748efbcd8f98fcc40ef5edae58"
Sep 17 21:06:10.561617 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.561557    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1f58b99e601386385b772c6dabcde54367d01748efbcd8f98fcc40ef5edae58"} err="failed to get container status \"f1f58b99e601386385b772c6dabcde54367d01748efbcd8f98fcc40ef5edae58\": rpc error: code = NotFound desc = could not find container \"f1f58b99e601386385b772c6dabcde54367d01748efbcd8f98fcc40ef5edae58\": container with ID starting with f1f58b99e601386385b772c6dabcde54367d01748efbcd8f98fcc40ef5edae58 not found: ID does not exist"
Sep 17 21:06:10.561617 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.561572    2566 scope.go:117] "RemoveContainer" containerID="c2a6c304167b8c6968fe073ded30d533edc6d049a90f4a16522fb079729946e7"
Sep 17 21:06:10.561806 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:06:10.561790    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2a6c304167b8c6968fe073ded30d533edc6d049a90f4a16522fb079729946e7\": container with ID starting with c2a6c304167b8c6968fe073ded30d533edc6d049a90f4a16522fb079729946e7 not found: ID does not exist" containerID="c2a6c304167b8c6968fe073ded30d533edc6d049a90f4a16522fb079729946e7"
Sep 17 21:06:10.561847 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.561811    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a6c304167b8c6968fe073ded30d533edc6d049a90f4a16522fb079729946e7"} err="failed to get container status \"c2a6c304167b8c6968fe073ded30d533edc6d049a90f4a16522fb079729946e7\": rpc error: code = NotFound desc = could not find container \"c2a6c304167b8c6968fe073ded30d533edc6d049a90f4a16522fb079729946e7\": container with ID starting with c2a6c304167b8c6968fe073ded30d533edc6d049a90f4a16522fb079729946e7 not found: ID does not exist"
Sep 17 21:06:10.561847 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.561824    2566 scope.go:117] "RemoveContainer" containerID="32598524d3fdd6e2788b0f7fa2d019f6b77dfcde22e31a2ee5a0ad412fca641e"
Sep 17 21:06:10.562053 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:06:10.562036    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32598524d3fdd6e2788b0f7fa2d019f6b77dfcde22e31a2ee5a0ad412fca641e\": container with ID starting with 32598524d3fdd6e2788b0f7fa2d019f6b77dfcde22e31a2ee5a0ad412fca641e not found: ID does not exist" containerID="32598524d3fdd6e2788b0f7fa2d019f6b77dfcde22e31a2ee5a0ad412fca641e"
Sep 17 21:06:10.562098 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.562058    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32598524d3fdd6e2788b0f7fa2d019f6b77dfcde22e31a2ee5a0ad412fca641e"} err="failed to get container status \"32598524d3fdd6e2788b0f7fa2d019f6b77dfcde22e31a2ee5a0ad412fca641e\": rpc error: code = NotFound desc = could not find container \"32598524d3fdd6e2788b0f7fa2d019f6b77dfcde22e31a2ee5a0ad412fca641e\": container with ID starting with 32598524d3fdd6e2788b0f7fa2d019f6b77dfcde22e31a2ee5a0ad412fca641e not found: ID does not exist"
Sep 17 21:06:10.567511 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.567492    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"]
Sep 17 21:06:10.572135 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:10.572116    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-hm692"]
Sep 17 21:06:12.396497 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.396474    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd"
Sep 17 21:06:12.459511 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.459443    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" path="/var/lib/kubelet/pods/77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9/volumes"
Sep 17 21:06:12.523808 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.523781    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08b521ce-377f-4f24-ba4a-95585aca48d5-cabundle-cert\") pod \"08b521ce-377f-4f24-ba4a-95585aca48d5\" (UID: \"08b521ce-377f-4f24-ba4a-95585aca48d5\") "
Sep 17 21:06:12.523976 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.523846    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08b521ce-377f-4f24-ba4a-95585aca48d5-proxy-tls\") pod \"08b521ce-377f-4f24-ba4a-95585aca48d5\" (UID: \"08b521ce-377f-4f24-ba4a-95585aca48d5\") "
Sep 17 21:06:12.523976 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.523903    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6qdl\" (UniqueName: \"kubernetes.io/projected/08b521ce-377f-4f24-ba4a-95585aca48d5-kube-api-access-j6qdl\") pod \"08b521ce-377f-4f24-ba4a-95585aca48d5\" (UID: \"08b521ce-377f-4f24-ba4a-95585aca48d5\") "
Sep 17 21:06:12.523976 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.523953    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08b521ce-377f-4f24-ba4a-95585aca48d5-kserve-provision-location\") pod \"08b521ce-377f-4f24-ba4a-95585aca48d5\" (UID: \"08b521ce-377f-4f24-ba4a-95585aca48d5\") "
Sep 17 21:06:12.524130 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.523985    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08b521ce-377f-4f24-ba4a-95585aca48d5-kube-rbac-proxy-sar-config\") pod \"08b521ce-377f-4f24-ba4a-95585aca48d5\" (UID: \"08b521ce-377f-4f24-ba4a-95585aca48d5\") "
Sep 17 21:06:12.524180 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.524140    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08b521ce-377f-4f24-ba4a-95585aca48d5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "08b521ce-377f-4f24-ba4a-95585aca48d5" (UID: "08b521ce-377f-4f24-ba4a-95585aca48d5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:06:12.524284 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.524261    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08b521ce-377f-4f24-ba4a-95585aca48d5-cabundle-cert\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:06:12.524369 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.524293    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b521ce-377f-4f24-ba4a-95585aca48d5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "08b521ce-377f-4f24-ba4a-95585aca48d5" (UID: "08b521ce-377f-4f24-ba4a-95585aca48d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:06:12.524417 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.524394    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08b521ce-377f-4f24-ba4a-95585aca48d5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "08b521ce-377f-4f24-ba4a-95585aca48d5" (UID: "08b521ce-377f-4f24-ba4a-95585aca48d5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:06:12.525980 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.525960    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08b521ce-377f-4f24-ba4a-95585aca48d5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "08b521ce-377f-4f24-ba4a-95585aca48d5" (UID: "08b521ce-377f-4f24-ba4a-95585aca48d5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:06:12.526074 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.526012    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b521ce-377f-4f24-ba4a-95585aca48d5-kube-api-access-j6qdl" (OuterVolumeSpecName: "kube-api-access-j6qdl") pod "08b521ce-377f-4f24-ba4a-95585aca48d5" (UID: "08b521ce-377f-4f24-ba4a-95585aca48d5"). InnerVolumeSpecName "kube-api-access-j6qdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:06:12.551576 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.551552    2566 generic.go:356] "Generic (PLEG): container finished" podID="08b521ce-377f-4f24-ba4a-95585aca48d5" containerID="7cd87444496f230ebceb924414eea393b2b8793d3fe165816d2d8a1b06628a1e" exitCode=0
Sep 17 21:06:12.551710 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.551627    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd" event={"ID":"08b521ce-377f-4f24-ba4a-95585aca48d5","Type":"ContainerDied","Data":"7cd87444496f230ebceb924414eea393b2b8793d3fe165816d2d8a1b06628a1e"}
Sep 17 21:06:12.551710 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.551626    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd"
Sep 17 21:06:12.551710 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.551679    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd" event={"ID":"08b521ce-377f-4f24-ba4a-95585aca48d5","Type":"ContainerDied","Data":"77aa688736eb3054abee4d67f409c04a666ed8186fd467488cdd789273d45066"}
Sep 17 21:06:12.551710 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.551704    2566 scope.go:117] "RemoveContainer" containerID="928aef75df7a2c1ba045a167fdf41e57f72816756d005836261c911b9fe18cde"
Sep 17 21:06:12.559842 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.559829    2566 scope.go:117] "RemoveContainer" containerID="7cd87444496f230ebceb924414eea393b2b8793d3fe165816d2d8a1b06628a1e"
Sep 17 21:06:12.567294 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.567280    2566 scope.go:117] "RemoveContainer" containerID="5573708bc3b434048d5b30be7e7c7ba686428affb32f99f1647436f5097cf302"
Sep 17 21:06:12.575155 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.575139    2566 scope.go:117] "RemoveContainer" containerID="928aef75df7a2c1ba045a167fdf41e57f72816756d005836261c911b9fe18cde"
Sep 17 21:06:12.575377 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.575355    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd"]
Sep 17 21:06:12.575495 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:06:12.575477    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"928aef75df7a2c1ba045a167fdf41e57f72816756d005836261c911b9fe18cde\": container with ID starting with 928aef75df7a2c1ba045a167fdf41e57f72816756d005836261c911b9fe18cde not found: ID does not exist" containerID="928aef75df7a2c1ba045a167fdf41e57f72816756d005836261c911b9fe18cde"
Sep 17 21:06:12.575540 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.575504    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"928aef75df7a2c1ba045a167fdf41e57f72816756d005836261c911b9fe18cde"} err="failed to get container status \"928aef75df7a2c1ba045a167fdf41e57f72816756d005836261c911b9fe18cde\": rpc error: code = NotFound desc = could not find container \"928aef75df7a2c1ba045a167fdf41e57f72816756d005836261c911b9fe18cde\": container with ID starting with 928aef75df7a2c1ba045a167fdf41e57f72816756d005836261c911b9fe18cde not found: ID does not exist"
Sep 17 21:06:12.575540 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.575520    2566 scope.go:117] "RemoveContainer" containerID="7cd87444496f230ebceb924414eea393b2b8793d3fe165816d2d8a1b06628a1e"
Sep 17 21:06:12.575776 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:06:12.575749    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cd87444496f230ebceb924414eea393b2b8793d3fe165816d2d8a1b06628a1e\": container with ID starting with 7cd87444496f230ebceb924414eea393b2b8793d3fe165816d2d8a1b06628a1e not found: ID does not exist" containerID="7cd87444496f230ebceb924414eea393b2b8793d3fe165816d2d8a1b06628a1e"
Sep 17 21:06:12.575832 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.575775    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cd87444496f230ebceb924414eea393b2b8793d3fe165816d2d8a1b06628a1e"} err="failed to get container status \"7cd87444496f230ebceb924414eea393b2b8793d3fe165816d2d8a1b06628a1e\": rpc error: code = NotFound desc = could not find container \"7cd87444496f230ebceb924414eea393b2b8793d3fe165816d2d8a1b06628a1e\": container with ID starting with 7cd87444496f230ebceb924414eea393b2b8793d3fe165816d2d8a1b06628a1e not found: ID does not exist"
Sep 17 21:06:12.575832 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.575792    2566 scope.go:117] "RemoveContainer" containerID="5573708bc3b434048d5b30be7e7c7ba686428affb32f99f1647436f5097cf302"
Sep 17 21:06:12.576055 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:06:12.576039    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5573708bc3b434048d5b30be7e7c7ba686428affb32f99f1647436f5097cf302\": container with ID starting with 5573708bc3b434048d5b30be7e7c7ba686428affb32f99f1647436f5097cf302 not found: ID does not exist" containerID="5573708bc3b434048d5b30be7e7c7ba686428affb32f99f1647436f5097cf302"
Sep 17 21:06:12.576103 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.576060    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5573708bc3b434048d5b30be7e7c7ba686428affb32f99f1647436f5097cf302"} err="failed to get container status \"5573708bc3b434048d5b30be7e7c7ba686428affb32f99f1647436f5097cf302\": rpc error: code = NotFound desc = could not find container \"5573708bc3b434048d5b30be7e7c7ba686428affb32f99f1647436f5097cf302\": container with ID starting with 5573708bc3b434048d5b30be7e7c7ba686428affb32f99f1647436f5097cf302 not found: ID does not exist"
Sep 17 21:06:12.578194 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.578175    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-ww6pd"]
Sep 17 21:06:12.624663 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.624642    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08b521ce-377f-4f24-ba4a-95585aca48d5-proxy-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:06:12.624663 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.624664    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6qdl\" (UniqueName: \"kubernetes.io/projected/08b521ce-377f-4f24-ba4a-95585aca48d5-kube-api-access-j6qdl\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:06:12.624823 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.624680    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08b521ce-377f-4f24-ba4a-95585aca48d5-kserve-provision-location\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:06:12.624823 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:12.624692    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08b521ce-377f-4f24-ba4a-95585aca48d5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:06:14.366803 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:14.366767    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5" podUID="432c0a3e-23a6-45ba-81f4-7cae8de3f677" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 21:06:14.459418 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:14.459378    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08b521ce-377f-4f24-ba4a-95585aca48d5" path="/var/lib/kubelet/pods/08b521ce-377f-4f24-ba4a-95585aca48d5/volumes"
Sep 17 21:06:16.272670 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:16.272630    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm" podUID="199305ac-35b9-4ccc-9ef4-7bb783c42118" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 21:06:24.366077 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:24.366035    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5" podUID="432c0a3e-23a6-45ba-81f4-7cae8de3f677" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 21:06:26.272107 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:26.272069    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm" podUID="199305ac-35b9-4ccc-9ef4-7bb783c42118" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 21:06:33.455384 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:33.455354    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm"
Sep 17 21:06:34.367369 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:06:34.367342    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5"
Sep 17 21:07:03.444979 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:03.444903    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm"]
Sep 17 21:07:03.445468 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:03.445154    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm" podUID="199305ac-35b9-4ccc-9ef4-7bb783c42118" containerName="kserve-container" containerID="cri-o://e21eb6e3f7526e220b91f8ffc2d375b30a661de17b098eb56f59d2178a9c3182" gracePeriod=30
Sep 17 21:07:03.445468 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:03.445262    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm" podUID="199305ac-35b9-4ccc-9ef4-7bb783c42118" containerName="kube-rbac-proxy" containerID="cri-o://7f0d0f2bd79e1332d65db51a22128b5564b5d777030a57badabc445dcd1f01e5" gracePeriod=30
Sep 17 21:07:03.455471 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:03.455437    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm" podUID="199305ac-35b9-4ccc-9ef4-7bb783c42118" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 21:07:03.495745 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:03.495718    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5"]
Sep 17 21:07:03.495966 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:03.495943    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5" podUID="432c0a3e-23a6-45ba-81f4-7cae8de3f677" containerName="kserve-container" containerID="cri-o://0ce45761da2b924c6425c41f781134c97dcaef76b2fc9997111f0c8ac99e5487" gracePeriod=30
Sep 17 21:07:03.496019 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:03.495974    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5" podUID="432c0a3e-23a6-45ba-81f4-7cae8de3f677" containerName="kube-rbac-proxy" containerID="cri-o://eb133b3d320148436296ce780efd7b1620b897af5519b12e078ddb577073b6e8" gracePeriod=30
Sep 17 21:07:03.754786 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:03.754757    2566 generic.go:356] "Generic (PLEG): container finished" podID="199305ac-35b9-4ccc-9ef4-7bb783c42118" containerID="7f0d0f2bd79e1332d65db51a22128b5564b5d777030a57badabc445dcd1f01e5" exitCode=2
Sep 17 21:07:03.754942 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:03.754834    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm" event={"ID":"199305ac-35b9-4ccc-9ef4-7bb783c42118","Type":"ContainerDied","Data":"7f0d0f2bd79e1332d65db51a22128b5564b5d777030a57badabc445dcd1f01e5"}
Sep 17 21:07:03.756787 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:03.756766    2566 generic.go:356] "Generic (PLEG): container finished" podID="432c0a3e-23a6-45ba-81f4-7cae8de3f677" containerID="eb133b3d320148436296ce780efd7b1620b897af5519b12e078ddb577073b6e8" exitCode=2
Sep 17 21:07:03.756897 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:03.756802    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5" event={"ID":"432c0a3e-23a6-45ba-81f4-7cae8de3f677","Type":"ContainerDied","Data":"eb133b3d320148436296ce780efd7b1620b897af5519b12e078ddb577073b6e8"}
Sep 17 21:07:04.360918 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:04.360877    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5" podUID="432c0a3e-23a6-45ba-81f4-7cae8de3f677" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 17 21:07:04.366262 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:04.366237    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5" podUID="432c0a3e-23a6-45ba-81f4-7cae8de3f677" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 21:07:06.266761 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:06.266716    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm" podUID="199305ac-35b9-4ccc-9ef4-7bb783c42118" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 17 21:07:07.330767 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.330744    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5"
Sep 17 21:07:07.460813 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.460732    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/432c0a3e-23a6-45ba-81f4-7cae8de3f677-kserve-provision-location\") pod \"432c0a3e-23a6-45ba-81f4-7cae8de3f677\" (UID: \"432c0a3e-23a6-45ba-81f4-7cae8de3f677\") "
Sep 17 21:07:07.460813 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.460762    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/432c0a3e-23a6-45ba-81f4-7cae8de3f677-cabundle-cert\") pod \"432c0a3e-23a6-45ba-81f4-7cae8de3f677\" (UID: \"432c0a3e-23a6-45ba-81f4-7cae8de3f677\") "
Sep 17 21:07:07.461046 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.460829    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/432c0a3e-23a6-45ba-81f4-7cae8de3f677-proxy-tls\") pod \"432c0a3e-23a6-45ba-81f4-7cae8de3f677\" (UID: \"432c0a3e-23a6-45ba-81f4-7cae8de3f677\") "
Sep 17 21:07:07.461046 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.460846    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/432c0a3e-23a6-45ba-81f4-7cae8de3f677-kube-rbac-proxy-sar-config\") pod \"432c0a3e-23a6-45ba-81f4-7cae8de3f677\" (UID: \"432c0a3e-23a6-45ba-81f4-7cae8de3f677\") "
Sep 17 21:07:07.461046 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.460866    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5q7k\" (UniqueName: \"kubernetes.io/projected/432c0a3e-23a6-45ba-81f4-7cae8de3f677-kube-api-access-j5q7k\") pod \"432c0a3e-23a6-45ba-81f4-7cae8de3f677\" (UID: \"432c0a3e-23a6-45ba-81f4-7cae8de3f677\") "
Sep 17 21:07:07.461196 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.461099    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432c0a3e-23a6-45ba-81f4-7cae8de3f677-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "432c0a3e-23a6-45ba-81f4-7cae8de3f677" (UID: "432c0a3e-23a6-45ba-81f4-7cae8de3f677"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:07:07.461196 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.461164    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/432c0a3e-23a6-45ba-81f4-7cae8de3f677-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "432c0a3e-23a6-45ba-81f4-7cae8de3f677" (UID: "432c0a3e-23a6-45ba-81f4-7cae8de3f677"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:07:07.461196 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.461181    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/432c0a3e-23a6-45ba-81f4-7cae8de3f677-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "432c0a3e-23a6-45ba-81f4-7cae8de3f677" (UID: "432c0a3e-23a6-45ba-81f4-7cae8de3f677"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:07:07.462953 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.462917    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432c0a3e-23a6-45ba-81f4-7cae8de3f677-kube-api-access-j5q7k" (OuterVolumeSpecName: "kube-api-access-j5q7k") pod "432c0a3e-23a6-45ba-81f4-7cae8de3f677" (UID: "432c0a3e-23a6-45ba-81f4-7cae8de3f677"). InnerVolumeSpecName "kube-api-access-j5q7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:07:07.463070 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.463013    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/432c0a3e-23a6-45ba-81f4-7cae8de3f677-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "432c0a3e-23a6-45ba-81f4-7cae8de3f677" (UID: "432c0a3e-23a6-45ba-81f4-7cae8de3f677"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:07:07.562013 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.561984    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/432c0a3e-23a6-45ba-81f4-7cae8de3f677-proxy-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:07:07.562013 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.562009    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/432c0a3e-23a6-45ba-81f4-7cae8de3f677-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:07:07.562178 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.562022    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5q7k\" (UniqueName: \"kubernetes.io/projected/432c0a3e-23a6-45ba-81f4-7cae8de3f677-kube-api-access-j5q7k\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:07:07.562178 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.562035    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/432c0a3e-23a6-45ba-81f4-7cae8de3f677-kserve-provision-location\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:07:07.562178 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.562047    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/432c0a3e-23a6-45ba-81f4-7cae8de3f677-cabundle-cert\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:07:07.773239 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.773193    2566 generic.go:356] "Generic (PLEG): container finished" podID="432c0a3e-23a6-45ba-81f4-7cae8de3f677" containerID="0ce45761da2b924c6425c41f781134c97dcaef76b2fc9997111f0c8ac99e5487" exitCode=0
Sep 17 21:07:07.773376 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.773263    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5"
Sep 17 21:07:07.773376 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.773276    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5" event={"ID":"432c0a3e-23a6-45ba-81f4-7cae8de3f677","Type":"ContainerDied","Data":"0ce45761da2b924c6425c41f781134c97dcaef76b2fc9997111f0c8ac99e5487"}
Sep 17 21:07:07.773376 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.773346    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5" event={"ID":"432c0a3e-23a6-45ba-81f4-7cae8de3f677","Type":"ContainerDied","Data":"bded7c4dccfe92025b1a0a80fb48776a23dd2a25b5ad0882f29687d56d3001c9"}
Sep 17 21:07:07.773495 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.773386    2566 scope.go:117] "RemoveContainer" containerID="eb133b3d320148436296ce780efd7b1620b897af5519b12e078ddb577073b6e8"
Sep 17 21:07:07.782549 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.782336    2566 scope.go:117] "RemoveContainer" containerID="0ce45761da2b924c6425c41f781134c97dcaef76b2fc9997111f0c8ac99e5487"
Sep 17 21:07:07.792098 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.792080    2566 scope.go:117] "RemoveContainer" containerID="fd7b5896ffb7b037fb3d3a063cee9b3631cde1a93a3cbb795b3e6760791e2413"
Sep 17 21:07:07.795469 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.795448    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5"]
Sep 17 21:07:07.799776 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.799751    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e3e0d-predictor-677b464669-9ckl5"]
Sep 17 21:07:07.800262 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.800245    2566 scope.go:117] "RemoveContainer" containerID="eb133b3d320148436296ce780efd7b1620b897af5519b12e078ddb577073b6e8"
Sep 17 21:07:07.800511 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:07:07.800494    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb133b3d320148436296ce780efd7b1620b897af5519b12e078ddb577073b6e8\": container with ID starting with eb133b3d320148436296ce780efd7b1620b897af5519b12e078ddb577073b6e8 not found: ID does not exist" containerID="eb133b3d320148436296ce780efd7b1620b897af5519b12e078ddb577073b6e8"
Sep 17 21:07:07.800558 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.800519    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb133b3d320148436296ce780efd7b1620b897af5519b12e078ddb577073b6e8"} err="failed to get container status \"eb133b3d320148436296ce780efd7b1620b897af5519b12e078ddb577073b6e8\": rpc error: code = NotFound desc = could not find container \"eb133b3d320148436296ce780efd7b1620b897af5519b12e078ddb577073b6e8\": container with ID starting with eb133b3d320148436296ce780efd7b1620b897af5519b12e078ddb577073b6e8 not found: ID does not exist"
Sep 17 21:07:07.800558 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.800533    2566 scope.go:117] "RemoveContainer" containerID="0ce45761da2b924c6425c41f781134c97dcaef76b2fc9997111f0c8ac99e5487"
Sep 17 21:07:07.800815 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:07:07.800792    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ce45761da2b924c6425c41f781134c97dcaef76b2fc9997111f0c8ac99e5487\": container with ID starting with 0ce45761da2b924c6425c41f781134c97dcaef76b2fc9997111f0c8ac99e5487 not found: ID does not exist" containerID="0ce45761da2b924c6425c41f781134c97dcaef76b2fc9997111f0c8ac99e5487"
Sep 17 21:07:07.800865 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.800826    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ce45761da2b924c6425c41f781134c97dcaef76b2fc9997111f0c8ac99e5487"} err="failed to get container status \"0ce45761da2b924c6425c41f781134c97dcaef76b2fc9997111f0c8ac99e5487\": rpc error: code = NotFound desc = could not find container \"0ce45761da2b924c6425c41f781134c97dcaef76b2fc9997111f0c8ac99e5487\": container with ID starting with 0ce45761da2b924c6425c41f781134c97dcaef76b2fc9997111f0c8ac99e5487 not found: ID does not exist"
Sep 17 21:07:07.800865 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.800846    2566 scope.go:117] "RemoveContainer" containerID="fd7b5896ffb7b037fb3d3a063cee9b3631cde1a93a3cbb795b3e6760791e2413"
Sep 17 21:07:07.801174 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:07:07.801158    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd7b5896ffb7b037fb3d3a063cee9b3631cde1a93a3cbb795b3e6760791e2413\": container with ID starting with fd7b5896ffb7b037fb3d3a063cee9b3631cde1a93a3cbb795b3e6760791e2413 not found: ID does not exist" containerID="fd7b5896ffb7b037fb3d3a063cee9b3631cde1a93a3cbb795b3e6760791e2413"
Sep 17 21:07:07.801245 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:07.801183    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd7b5896ffb7b037fb3d3a063cee9b3631cde1a93a3cbb795b3e6760791e2413"} err="failed to get container status \"fd7b5896ffb7b037fb3d3a063cee9b3631cde1a93a3cbb795b3e6760791e2413\": rpc error: code = NotFound desc = could not find container \"fd7b5896ffb7b037fb3d3a063cee9b3631cde1a93a3cbb795b3e6760791e2413\": container with ID starting with fd7b5896ffb7b037fb3d3a063cee9b3631cde1a93a3cbb795b3e6760791e2413 not found: ID does not exist"
Sep 17 21:07:08.172728 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.172705    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm"
Sep 17 21:07:08.267877 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.267851    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/199305ac-35b9-4ccc-9ef4-7bb783c42118-kube-rbac-proxy-sar-config\") pod \"199305ac-35b9-4ccc-9ef4-7bb783c42118\" (UID: \"199305ac-35b9-4ccc-9ef4-7bb783c42118\") "
Sep 17 21:07:08.268012 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.267882    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/199305ac-35b9-4ccc-9ef4-7bb783c42118-kserve-provision-location\") pod \"199305ac-35b9-4ccc-9ef4-7bb783c42118\" (UID: \"199305ac-35b9-4ccc-9ef4-7bb783c42118\") "
Sep 17 21:07:08.268012 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.267915    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/199305ac-35b9-4ccc-9ef4-7bb783c42118-proxy-tls\") pod \"199305ac-35b9-4ccc-9ef4-7bb783c42118\" (UID: \"199305ac-35b9-4ccc-9ef4-7bb783c42118\") "
Sep 17 21:07:08.268012 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.267961    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/199305ac-35b9-4ccc-9ef4-7bb783c42118-cabundle-cert\") pod \"199305ac-35b9-4ccc-9ef4-7bb783c42118\" (UID: \"199305ac-35b9-4ccc-9ef4-7bb783c42118\") "
Sep 17 21:07:08.268130 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.268011    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtpkh\" (UniqueName: \"kubernetes.io/projected/199305ac-35b9-4ccc-9ef4-7bb783c42118-kube-api-access-wtpkh\") pod \"199305ac-35b9-4ccc-9ef4-7bb783c42118\" (UID: \"199305ac-35b9-4ccc-9ef4-7bb783c42118\") "
Sep 17 21:07:08.268250 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.268204    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/199305ac-35b9-4ccc-9ef4-7bb783c42118-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "199305ac-35b9-4ccc-9ef4-7bb783c42118" (UID: "199305ac-35b9-4ccc-9ef4-7bb783c42118"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:07:08.268327 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.268247    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/199305ac-35b9-4ccc-9ef4-7bb783c42118-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "199305ac-35b9-4ccc-9ef4-7bb783c42118" (UID: "199305ac-35b9-4ccc-9ef4-7bb783c42118"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:07:08.268327 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.268311    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/199305ac-35b9-4ccc-9ef4-7bb783c42118-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "199305ac-35b9-4ccc-9ef4-7bb783c42118" (UID: "199305ac-35b9-4ccc-9ef4-7bb783c42118"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:07:08.269955 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.269929    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/199305ac-35b9-4ccc-9ef4-7bb783c42118-kube-api-access-wtpkh" (OuterVolumeSpecName: "kube-api-access-wtpkh") pod "199305ac-35b9-4ccc-9ef4-7bb783c42118" (UID: "199305ac-35b9-4ccc-9ef4-7bb783c42118"). InnerVolumeSpecName "kube-api-access-wtpkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:07:08.270054 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.269970    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/199305ac-35b9-4ccc-9ef4-7bb783c42118-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "199305ac-35b9-4ccc-9ef4-7bb783c42118" (UID: "199305ac-35b9-4ccc-9ef4-7bb783c42118"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:07:08.368985 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.368933    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtpkh\" (UniqueName: \"kubernetes.io/projected/199305ac-35b9-4ccc-9ef4-7bb783c42118-kube-api-access-wtpkh\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:07:08.368985 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.368951    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/199305ac-35b9-4ccc-9ef4-7bb783c42118-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:07:08.368985 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.368960    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/199305ac-35b9-4ccc-9ef4-7bb783c42118-kserve-provision-location\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:07:08.368985 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.368968    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/199305ac-35b9-4ccc-9ef4-7bb783c42118-proxy-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:07:08.368985 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.368976    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/199305ac-35b9-4ccc-9ef4-7bb783c42118-cabundle-cert\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:07:08.459371 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.459344    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="432c0a3e-23a6-45ba-81f4-7cae8de3f677" path="/var/lib/kubelet/pods/432c0a3e-23a6-45ba-81f4-7cae8de3f677/volumes"
Sep 17 21:07:08.779536 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.779501    2566 generic.go:356] "Generic (PLEG): container finished" podID="199305ac-35b9-4ccc-9ef4-7bb783c42118" containerID="e21eb6e3f7526e220b91f8ffc2d375b30a661de17b098eb56f59d2178a9c3182" exitCode=0
Sep 17 21:07:08.779714 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.779562    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm"
Sep 17 21:07:08.779714 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.779586    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm" event={"ID":"199305ac-35b9-4ccc-9ef4-7bb783c42118","Type":"ContainerDied","Data":"e21eb6e3f7526e220b91f8ffc2d375b30a661de17b098eb56f59d2178a9c3182"}
Sep 17 21:07:08.779714 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.779669    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm" event={"ID":"199305ac-35b9-4ccc-9ef4-7bb783c42118","Type":"ContainerDied","Data":"e062930eccf1518f78b843699a5968d271e3a65ff2730abd1397f124fecfbdd4"}
Sep 17 21:07:08.779714 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.779710    2566 scope.go:117] "RemoveContainer" containerID="7f0d0f2bd79e1332d65db51a22128b5564b5d777030a57badabc445dcd1f01e5"
Sep 17 21:07:08.792483 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.792464    2566 scope.go:117] "RemoveContainer" containerID="e21eb6e3f7526e220b91f8ffc2d375b30a661de17b098eb56f59d2178a9c3182"
Sep 17 21:07:08.799540 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.799522    2566 scope.go:117] "RemoveContainer" containerID="79e14833b1ce1f31fb97edd02b4a75ea1ab11165ec2ac2e50799d15333446d25"
Sep 17 21:07:08.807430 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.807413    2566 scope.go:117] "RemoveContainer" containerID="7f0d0f2bd79e1332d65db51a22128b5564b5d777030a57badabc445dcd1f01e5"
Sep 17 21:07:08.807569 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.807546    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm"]
Sep 17 21:07:08.807713 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:07:08.807694    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f0d0f2bd79e1332d65db51a22128b5564b5d777030a57badabc445dcd1f01e5\": container with ID starting with 7f0d0f2bd79e1332d65db51a22128b5564b5d777030a57badabc445dcd1f01e5 not found: ID does not exist" containerID="7f0d0f2bd79e1332d65db51a22128b5564b5d777030a57badabc445dcd1f01e5"
Sep 17 21:07:08.807759 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.807722    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f0d0f2bd79e1332d65db51a22128b5564b5d777030a57badabc445dcd1f01e5"} err="failed to get container status \"7f0d0f2bd79e1332d65db51a22128b5564b5d777030a57badabc445dcd1f01e5\": rpc error: code = NotFound desc = could not find container \"7f0d0f2bd79e1332d65db51a22128b5564b5d777030a57badabc445dcd1f01e5\": container with ID starting with 7f0d0f2bd79e1332d65db51a22128b5564b5d777030a57badabc445dcd1f01e5 not found: ID does not exist"
Sep 17 21:07:08.807759 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.807739    2566 scope.go:117] "RemoveContainer" containerID="e21eb6e3f7526e220b91f8ffc2d375b30a661de17b098eb56f59d2178a9c3182"
Sep 17 21:07:08.807998 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:07:08.807981    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e21eb6e3f7526e220b91f8ffc2d375b30a661de17b098eb56f59d2178a9c3182\": container with ID starting with e21eb6e3f7526e220b91f8ffc2d375b30a661de17b098eb56f59d2178a9c3182 not found: ID does not exist" containerID="e21eb6e3f7526e220b91f8ffc2d375b30a661de17b098eb56f59d2178a9c3182"
Sep 17 21:07:08.808040 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.808004    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e21eb6e3f7526e220b91f8ffc2d375b30a661de17b098eb56f59d2178a9c3182"} err="failed to get container status \"e21eb6e3f7526e220b91f8ffc2d375b30a661de17b098eb56f59d2178a9c3182\": rpc error: code = NotFound desc = could not find container \"e21eb6e3f7526e220b91f8ffc2d375b30a661de17b098eb56f59d2178a9c3182\": container with ID starting with e21eb6e3f7526e220b91f8ffc2d375b30a661de17b098eb56f59d2178a9c3182 not found: ID does not exist"
Sep 17 21:07:08.808040 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.808019    2566 scope.go:117] "RemoveContainer" containerID="79e14833b1ce1f31fb97edd02b4a75ea1ab11165ec2ac2e50799d15333446d25"
Sep 17 21:07:08.808306 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:07:08.808283    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79e14833b1ce1f31fb97edd02b4a75ea1ab11165ec2ac2e50799d15333446d25\": container with ID starting with 79e14833b1ce1f31fb97edd02b4a75ea1ab11165ec2ac2e50799d15333446d25 not found: ID does not exist" containerID="79e14833b1ce1f31fb97edd02b4a75ea1ab11165ec2ac2e50799d15333446d25"
Sep 17 21:07:08.808354 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.808320    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79e14833b1ce1f31fb97edd02b4a75ea1ab11165ec2ac2e50799d15333446d25"} err="failed to get container status \"79e14833b1ce1f31fb97edd02b4a75ea1ab11165ec2ac2e50799d15333446d25\": rpc error: code = NotFound desc = could not find container \"79e14833b1ce1f31fb97edd02b4a75ea1ab11165ec2ac2e50799d15333446d25\": container with ID starting with 79e14833b1ce1f31fb97edd02b4a75ea1ab11165ec2ac2e50799d15333446d25 not found: ID does not exist"
Sep 17 21:07:08.812494 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:08.812473    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e3e0d-predictor-d6fc66f56-bmzxm"]
Sep 17 21:07:10.460601 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:10.460569    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="199305ac-35b9-4ccc-9ef4-7bb783c42118" path="/var/lib/kubelet/pods/199305ac-35b9-4ccc-9ef4-7bb783c42118/volumes"
Sep 17 21:07:13.533547 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.533513    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"]
Sep 17 21:07:13.533997 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.533908    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b521ce-377f-4f24-ba4a-95585aca48d5" containerName="storage-initializer"
Sep 17 21:07:13.533997 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.533926    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b521ce-377f-4f24-ba4a-95585aca48d5" containerName="storage-initializer"
Sep 17 21:07:13.533997 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.533941    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b521ce-377f-4f24-ba4a-95585aca48d5" containerName="kserve-container"
Sep 17 21:07:13.533997 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.533949    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b521ce-377f-4f24-ba4a-95585aca48d5" containerName="kserve-container"
Sep 17 21:07:13.533997 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.533961    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b521ce-377f-4f24-ba4a-95585aca48d5" containerName="kube-rbac-proxy"
Sep 17 21:07:13.533997 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.533969    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b521ce-377f-4f24-ba4a-95585aca48d5" containerName="kube-rbac-proxy"
Sep 17 21:07:13.533997 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.533979    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" containerName="kserve-container"
Sep 17 21:07:13.533997 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.533987    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" containerName="kserve-container"
Sep 17 21:07:13.533997 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.533997    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="432c0a3e-23a6-45ba-81f4-7cae8de3f677" containerName="storage-initializer"
Sep 17 21:07:13.534505 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.534005    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="432c0a3e-23a6-45ba-81f4-7cae8de3f677" containerName="storage-initializer"
Sep 17 21:07:13.534505 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.534015    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="199305ac-35b9-4ccc-9ef4-7bb783c42118" containerName="kserve-container"
Sep 17 21:07:13.534505 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.534023    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="199305ac-35b9-4ccc-9ef4-7bb783c42118" containerName="kserve-container"
Sep 17 21:07:13.534505 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.534039    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" containerName="kube-rbac-proxy"
Sep 17 21:07:13.534505 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.534047    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" containerName="kube-rbac-proxy"
Sep 17 21:07:13.534505 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.534062    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="199305ac-35b9-4ccc-9ef4-7bb783c42118" containerName="storage-initializer"
Sep 17 21:07:13.534505 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.534070    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="199305ac-35b9-4ccc-9ef4-7bb783c42118" containerName="storage-initializer"
Sep 17 21:07:13.534505 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.534082    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" containerName="storage-initializer"
Sep 17 21:07:13.534505 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.534090    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" containerName="storage-initializer"
Sep 17 21:07:13.534505 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.534104    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="199305ac-35b9-4ccc-9ef4-7bb783c42118" containerName="kube-rbac-proxy"
Sep 17 21:07:13.534505 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.534113    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="199305ac-35b9-4ccc-9ef4-7bb783c42118" containerName="kube-rbac-proxy"
Sep 17 21:07:13.534505 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.534123    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="432c0a3e-23a6-45ba-81f4-7cae8de3f677" containerName="kserve-container"
Sep 17 21:07:13.534505 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.534130    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="432c0a3e-23a6-45ba-81f4-7cae8de3f677" containerName="kserve-container"
Sep 17 21:07:13.534505 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.534143    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="432c0a3e-23a6-45ba-81f4-7cae8de3f677" containerName="kube-rbac-proxy"
Sep 17 21:07:13.534505 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.534151    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="432c0a3e-23a6-45ba-81f4-7cae8de3f677" containerName="kube-rbac-proxy"
Sep 17 21:07:13.534505 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.534241    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="08b521ce-377f-4f24-ba4a-95585aca48d5" containerName="kserve-container"
Sep 17 21:07:13.534505 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.534252    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="199305ac-35b9-4ccc-9ef4-7bb783c42118" containerName="kserve-container"
Sep 17 21:07:13.534505 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.534264    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="199305ac-35b9-4ccc-9ef4-7bb783c42118" containerName="kube-rbac-proxy"
Sep 17 21:07:13.534505 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.534273    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" containerName="kserve-container"
Sep 17 21:07:13.534505 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.534284    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="08b521ce-377f-4f24-ba4a-95585aca48d5" containerName="kube-rbac-proxy"
Sep 17 21:07:13.534505 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.534293    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="432c0a3e-23a6-45ba-81f4-7cae8de3f677" containerName="kserve-container"
Sep 17 21:07:13.534505 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.534302    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="432c0a3e-23a6-45ba-81f4-7cae8de3f677" containerName="kube-rbac-proxy"
Sep 17 21:07:13.534505 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.534310    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="77dcbaf7-6ebd-45c9-87c0-c1fb3ab31cc9" containerName="kube-rbac-proxy"
Sep 17 21:07:13.539310 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.539291    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"
Sep 17 21:07:13.541917 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.541898    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-8d8d6-predictor-serving-cert\""
Sep 17 21:07:13.542043 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.541980    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5dj6n\""
Sep 17 21:07:13.542111 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.542093    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 21:07:13.542179 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.542163    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-8d8d6-kube-rbac-proxy-sar-config\""
Sep 17 21:07:13.542243 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.542178    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 21:07:13.542243 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.542180    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 21:07:13.542922 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.542909    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:07:13.547421 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.547400    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"]
Sep 17 21:07:13.606318 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.606292    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w\" (UID: \"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"
Sep 17 21:07:13.606437 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.606326    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-cabundle-cert\") pod \"isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w\" (UID: \"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"
Sep 17 21:07:13.606437 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.606351    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkx89\" (UniqueName: \"kubernetes.io/projected/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-kube-api-access-zkx89\") pod \"isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w\" (UID: \"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"
Sep 17 21:07:13.606437 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.606379    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-kserve-provision-location\") pod \"isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w\" (UID: \"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"
Sep 17 21:07:13.606437 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.606411    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-proxy-tls\") pod \"isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w\" (UID: \"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"
Sep 17 21:07:13.707547 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.707512    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w\" (UID: \"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"
Sep 17 21:07:13.707700 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.707554    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-cabundle-cert\") pod \"isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w\" (UID: \"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"
Sep 17 21:07:13.707700 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.707589    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkx89\" (UniqueName: \"kubernetes.io/projected/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-kube-api-access-zkx89\") pod \"isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w\" (UID: \"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"
Sep 17 21:07:13.707700 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.707614    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-kserve-provision-location\") pod \"isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w\" (UID: \"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"
Sep 17 21:07:13.707700 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.707666    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-proxy-tls\") pod \"isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w\" (UID: \"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"
Sep 17 21:07:13.707899 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:07:13.707835    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-serving-cert: secret "isvc-logger-raw-8d8d6-predictor-serving-cert" not found
Sep 17 21:07:13.707962 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:07:13.707925    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-proxy-tls podName:46cdee62-f1b1-4a8b-bd3c-0717fb883fe9 nodeName:}" failed. No retries permitted until 2026-09-17 21:07:14.207903153 +0000 UTC m=+1075.818636363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-proxy-tls") pod "isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" (UID: "46cdee62-f1b1-4a8b-bd3c-0717fb883fe9") : secret "isvc-logger-raw-8d8d6-predictor-serving-cert" not found
Sep 17 21:07:13.708079 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.708058    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-kserve-provision-location\") pod \"isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w\" (UID: \"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"
Sep 17 21:07:13.708248 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.708199    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w\" (UID: \"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"
Sep 17 21:07:13.708343 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.708299    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-cabundle-cert\") pod \"isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w\" (UID: \"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"
Sep 17 21:07:13.717233 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:13.717194    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkx89\" (UniqueName: \"kubernetes.io/projected/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-kube-api-access-zkx89\") pod \"isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w\" (UID: \"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"
Sep 17 21:07:14.211091 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:14.211057    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-proxy-tls\") pod \"isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w\" (UID: \"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"
Sep 17 21:07:14.213397 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:14.213377    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-proxy-tls\") pod \"isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w\" (UID: \"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"
Sep 17 21:07:14.450336 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:14.450310    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"
Sep 17 21:07:14.726501 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:14.726473    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"]
Sep 17 21:07:14.728527 ip-10-0-138-140 kubenswrapper[2566]: W0917 21:07:14.728490    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46cdee62_f1b1_4a8b_bd3c_0717fb883fe9.slice/crio-2ff80ef14580fcae01b5bb4b61c13845689844253b33037808fc74fa566a881e WatchSource:0}: Error finding container 2ff80ef14580fcae01b5bb4b61c13845689844253b33037808fc74fa566a881e: Status 404 returned error can't find the container with id 2ff80ef14580fcae01b5bb4b61c13845689844253b33037808fc74fa566a881e
Sep 17 21:07:14.806301 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:14.806271    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" event={"ID":"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9","Type":"ContainerStarted","Data":"bd48be16c55a2539b5af500393af0b7e7d7acf89add20f69587b49838c215d0a"}
Sep 17 21:07:14.806426 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:14.806345    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" event={"ID":"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9","Type":"ContainerStarted","Data":"2ff80ef14580fcae01b5bb4b61c13845689844253b33037808fc74fa566a881e"}
Sep 17 21:07:18.822702 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:18.822669    2566 generic.go:356] "Generic (PLEG): container finished" podID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerID="bd48be16c55a2539b5af500393af0b7e7d7acf89add20f69587b49838c215d0a" exitCode=0
Sep 17 21:07:18.823048 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:18.822743    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" event={"ID":"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9","Type":"ContainerDied","Data":"bd48be16c55a2539b5af500393af0b7e7d7acf89add20f69587b49838c215d0a"}
Sep 17 21:07:19.828842 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:19.828812    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" event={"ID":"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9","Type":"ContainerStarted","Data":"0ba165e0e0738b6ef8212ef7f0d79a8384e2c80624b40251404536cceb15c1cc"}
Sep 17 21:07:19.829310 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:19.828874    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" event={"ID":"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9","Type":"ContainerStarted","Data":"0b2774f2aa249db63c52d1ce9a8c6af8b63663893d944e3bd459cb44db0c9759"}
Sep 17 21:07:19.829310 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:19.828901    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" event={"ID":"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9","Type":"ContainerStarted","Data":"6d55c9765ef22826a335df0f5a984731a79a4febd352f575af945f16d157bc4b"}
Sep 17 21:07:19.829310 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:19.829091    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"
Sep 17 21:07:19.854619 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:19.854564    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podStartSLOduration=3.18556638 podStartE2EDuration="6.854546786s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.668980406s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:07:19.852937595 +0000 UTC m=+1081.463670824" watchObservedRunningTime="2026-09-17 21:07:19.854546786 +0000 UTC m=+1081.465280016"
Sep 17 21:07:20.832904 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:20.832872    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"
Sep 17 21:07:20.833295 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:20.832931    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"
Sep 17 21:07:20.834231 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:20.834178    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 21:07:20.834927 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:20.834895    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:07:21.837100 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:21.837058    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 21:07:21.837504 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:21.837418    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:07:26.841174 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:26.841149    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"
Sep 17 21:07:26.841633 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:26.841595    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 21:07:26.841969 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:26.841944    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:07:36.841785 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:36.841745    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 21:07:36.842348 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:36.842203    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:07:46.842400 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:46.842359    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 21:07:46.842890 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:46.842864    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:07:56.841949 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:56.841904    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 21:07:56.842424 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:07:56.842387    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:08:02.988412 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:02.988380    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz"]
Sep 17 21:08:02.991965 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:02.991943    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz"
Sep 17 21:08:02.994382 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:02.994358    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-b61317-predictor-serving-cert\""
Sep 17 21:08:02.994382 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:02.994370    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-b61317-kube-rbac-proxy-sar-config\""
Sep 17 21:08:02.994717 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:02.994702    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-b61317-dockercfg-2nstt\""
Sep 17 21:08:02.994784 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:02.994718    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-b61317\""
Sep 17 21:08:03.001593 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:03.001571    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz"]
Sep 17 21:08:03.090827 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:03.090793    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5c7b2df-6d85-45f5-9190-d81cf88321e6-cabundle-cert\") pod \"isvc-secondary-b61317-predictor-6475c545c4-dk2kz\" (UID: \"b5c7b2df-6d85-45f5-9190-d81cf88321e6\") " pod="kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz"
Sep 17 21:08:03.091006 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:03.090844    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5c7b2df-6d85-45f5-9190-d81cf88321e6-proxy-tls\") pod \"isvc-secondary-b61317-predictor-6475c545c4-dk2kz\" (UID: \"b5c7b2df-6d85-45f5-9190-d81cf88321e6\") " pod="kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz"
Sep 17 21:08:03.091006 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:03.090877    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5c7b2df-6d85-45f5-9190-d81cf88321e6-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-b61317-predictor-6475c545c4-dk2kz\" (UID: \"b5c7b2df-6d85-45f5-9190-d81cf88321e6\") " pod="kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz"
Sep 17 21:08:03.091006 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:03.090946    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8vgs\" (UniqueName: \"kubernetes.io/projected/b5c7b2df-6d85-45f5-9190-d81cf88321e6-kube-api-access-r8vgs\") pod \"isvc-secondary-b61317-predictor-6475c545c4-dk2kz\" (UID: \"b5c7b2df-6d85-45f5-9190-d81cf88321e6\") " pod="kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz"
Sep 17 21:08:03.091006 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:03.090979    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5c7b2df-6d85-45f5-9190-d81cf88321e6-kserve-provision-location\") pod \"isvc-secondary-b61317-predictor-6475c545c4-dk2kz\" (UID: \"b5c7b2df-6d85-45f5-9190-d81cf88321e6\") " pod="kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz"
Sep 17 21:08:03.192219 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:03.192189    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5c7b2df-6d85-45f5-9190-d81cf88321e6-cabundle-cert\") pod \"isvc-secondary-b61317-predictor-6475c545c4-dk2kz\" (UID: \"b5c7b2df-6d85-45f5-9190-d81cf88321e6\") " pod="kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz"
Sep 17 21:08:03.192377 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:03.192247    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5c7b2df-6d85-45f5-9190-d81cf88321e6-proxy-tls\") pod \"isvc-secondary-b61317-predictor-6475c545c4-dk2kz\" (UID: \"b5c7b2df-6d85-45f5-9190-d81cf88321e6\") " pod="kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz"
Sep 17 21:08:03.192377 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:03.192269    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5c7b2df-6d85-45f5-9190-d81cf88321e6-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-b61317-predictor-6475c545c4-dk2kz\" (UID: \"b5c7b2df-6d85-45f5-9190-d81cf88321e6\") " pod="kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz"
Sep 17 21:08:03.192377 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:03.192307    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8vgs\" (UniqueName: \"kubernetes.io/projected/b5c7b2df-6d85-45f5-9190-d81cf88321e6-kube-api-access-r8vgs\") pod \"isvc-secondary-b61317-predictor-6475c545c4-dk2kz\" (UID: \"b5c7b2df-6d85-45f5-9190-d81cf88321e6\") " pod="kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz"
Sep 17 21:08:03.192377 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:03.192328    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5c7b2df-6d85-45f5-9190-d81cf88321e6-kserve-provision-location\") pod \"isvc-secondary-b61317-predictor-6475c545c4-dk2kz\" (UID: \"b5c7b2df-6d85-45f5-9190-d81cf88321e6\") " pod="kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz"
Sep 17 21:08:03.192577 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:08:03.192405    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-b61317-predictor-serving-cert: secret "isvc-secondary-b61317-predictor-serving-cert" not found
Sep 17 21:08:03.192577 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:08:03.192472    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5c7b2df-6d85-45f5-9190-d81cf88321e6-proxy-tls podName:b5c7b2df-6d85-45f5-9190-d81cf88321e6 nodeName:}" failed. No retries permitted until 2026-09-17 21:08:03.692451234 +0000 UTC m=+1125.303184463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b5c7b2df-6d85-45f5-9190-d81cf88321e6-proxy-tls") pod "isvc-secondary-b61317-predictor-6475c545c4-dk2kz" (UID: "b5c7b2df-6d85-45f5-9190-d81cf88321e6") : secret "isvc-secondary-b61317-predictor-serving-cert" not found
Sep 17 21:08:03.192689 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:03.192662    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5c7b2df-6d85-45f5-9190-d81cf88321e6-kserve-provision-location\") pod \"isvc-secondary-b61317-predictor-6475c545c4-dk2kz\" (UID: \"b5c7b2df-6d85-45f5-9190-d81cf88321e6\") " pod="kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz"
Sep 17 21:08:03.192947 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:03.192928    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5c7b2df-6d85-45f5-9190-d81cf88321e6-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-b61317-predictor-6475c545c4-dk2kz\" (UID: \"b5c7b2df-6d85-45f5-9190-d81cf88321e6\") " pod="kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz"
Sep 17 21:08:03.193026 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:03.192991    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5c7b2df-6d85-45f5-9190-d81cf88321e6-cabundle-cert\") pod \"isvc-secondary-b61317-predictor-6475c545c4-dk2kz\" (UID: \"b5c7b2df-6d85-45f5-9190-d81cf88321e6\") " pod="kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz"
Sep 17 21:08:03.200974 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:03.200952    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8vgs\" (UniqueName: \"kubernetes.io/projected/b5c7b2df-6d85-45f5-9190-d81cf88321e6-kube-api-access-r8vgs\") pod \"isvc-secondary-b61317-predictor-6475c545c4-dk2kz\" (UID: \"b5c7b2df-6d85-45f5-9190-d81cf88321e6\") " pod="kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz"
Sep 17 21:08:03.696936 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:03.696895    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5c7b2df-6d85-45f5-9190-d81cf88321e6-proxy-tls\") pod \"isvc-secondary-b61317-predictor-6475c545c4-dk2kz\" (UID: \"b5c7b2df-6d85-45f5-9190-d81cf88321e6\") " pod="kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz"
Sep 17 21:08:03.699305 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:03.699285    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5c7b2df-6d85-45f5-9190-d81cf88321e6-proxy-tls\") pod \"isvc-secondary-b61317-predictor-6475c545c4-dk2kz\" (UID: \"b5c7b2df-6d85-45f5-9190-d81cf88321e6\") " pod="kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz"
Sep 17 21:08:03.903307 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:03.903270    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz"
Sep 17 21:08:04.173417 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:04.173381    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz"]
Sep 17 21:08:04.176700 ip-10-0-138-140 kubenswrapper[2566]: W0917 21:08:04.176670    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5c7b2df_6d85_45f5_9190_d81cf88321e6.slice/crio-f9826a9001eb57d8091746fc211f9cb3c5d92b2f7a67535c784932e6816ba554 WatchSource:0}: Error finding container f9826a9001eb57d8091746fc211f9cb3c5d92b2f7a67535c784932e6816ba554: Status 404 returned error can't find the container with id f9826a9001eb57d8091746fc211f9cb3c5d92b2f7a67535c784932e6816ba554
Sep 17 21:08:05.012565 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:05.012530    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz" event={"ID":"b5c7b2df-6d85-45f5-9190-d81cf88321e6","Type":"ContainerStarted","Data":"07784e9c56c0fb3fd964278f0cb15aadc16d80ff34ba3cc44093c91284e2d8fe"}
Sep 17 21:08:05.012730 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:05.012590    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz" event={"ID":"b5c7b2df-6d85-45f5-9190-d81cf88321e6","Type":"ContainerStarted","Data":"f9826a9001eb57d8091746fc211f9cb3c5d92b2f7a67535c784932e6816ba554"}
Sep 17 21:08:06.841754 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:06.841713    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 21:08:06.842132 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:06.841949    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:08:08.027270 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:08.027244    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b61317-predictor-6475c545c4-dk2kz_b5c7b2df-6d85-45f5-9190-d81cf88321e6/storage-initializer/0.log"
Sep 17 21:08:08.027829 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:08.027280    2566 generic.go:356] "Generic (PLEG): container finished" podID="b5c7b2df-6d85-45f5-9190-d81cf88321e6" containerID="07784e9c56c0fb3fd964278f0cb15aadc16d80ff34ba3cc44093c91284e2d8fe" exitCode=1
Sep 17 21:08:08.027829 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:08.027366    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz" event={"ID":"b5c7b2df-6d85-45f5-9190-d81cf88321e6","Type":"ContainerDied","Data":"07784e9c56c0fb3fd964278f0cb15aadc16d80ff34ba3cc44093c91284e2d8fe"}
Sep 17 21:08:09.033383 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:09.033357    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b61317-predictor-6475c545c4-dk2kz_b5c7b2df-6d85-45f5-9190-d81cf88321e6/storage-initializer/0.log"
Sep 17 21:08:09.033773 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:09.033443    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz" event={"ID":"b5c7b2df-6d85-45f5-9190-d81cf88321e6","Type":"ContainerStarted","Data":"d848f5e44bc5d359ee07bde3d963947bf3804d9766bc3cdb8ba6633f16c7b6b2"}
Sep 17 21:08:12.046902 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:12.046878    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b61317-predictor-6475c545c4-dk2kz_b5c7b2df-6d85-45f5-9190-d81cf88321e6/storage-initializer/1.log"
Sep 17 21:08:12.047323 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:12.047253    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b61317-predictor-6475c545c4-dk2kz_b5c7b2df-6d85-45f5-9190-d81cf88321e6/storage-initializer/0.log"
Sep 17 21:08:12.047323 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:12.047288    2566 generic.go:356] "Generic (PLEG): container finished" podID="b5c7b2df-6d85-45f5-9190-d81cf88321e6" containerID="d848f5e44bc5d359ee07bde3d963947bf3804d9766bc3cdb8ba6633f16c7b6b2" exitCode=1
Sep 17 21:08:12.047397 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:12.047320    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz" event={"ID":"b5c7b2df-6d85-45f5-9190-d81cf88321e6","Type":"ContainerDied","Data":"d848f5e44bc5d359ee07bde3d963947bf3804d9766bc3cdb8ba6633f16c7b6b2"}
Sep 17 21:08:12.047397 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:12.047380    2566 scope.go:117] "RemoveContainer" containerID="07784e9c56c0fb3fd964278f0cb15aadc16d80ff34ba3cc44093c91284e2d8fe"
Sep 17 21:08:12.047789 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:12.047770    2566 scope.go:117] "RemoveContainer" containerID="07784e9c56c0fb3fd964278f0cb15aadc16d80ff34ba3cc44093c91284e2d8fe"
Sep 17 21:08:12.057821 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:08:12.057802    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-b61317-predictor-6475c545c4-dk2kz_kserve-ci-e2e-test(b5c7b2df-6d85-45f5-9190-d81cf88321e6)\"" pod="kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz" podUID="b5c7b2df-6d85-45f5-9190-d81cf88321e6"
Sep 17 21:08:12.058034 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:08:12.057998    2566 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-b61317-predictor-6475c545c4-dk2kz_kserve-ci-e2e-test_b5c7b2df-6d85-45f5-9190-d81cf88321e6_0 in pod sandbox f9826a9001eb57d8091746fc211f9cb3c5d92b2f7a67535c784932e6816ba554 from index: no such id: '07784e9c56c0fb3fd964278f0cb15aadc16d80ff34ba3cc44093c91284e2d8fe'" containerID="07784e9c56c0fb3fd964278f0cb15aadc16d80ff34ba3cc44093c91284e2d8fe"
Sep 17 21:08:12.058081 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:12.058047    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07784e9c56c0fb3fd964278f0cb15aadc16d80ff34ba3cc44093c91284e2d8fe"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-b61317-predictor-6475c545c4-dk2kz_kserve-ci-e2e-test_b5c7b2df-6d85-45f5-9190-d81cf88321e6_0 in pod sandbox f9826a9001eb57d8091746fc211f9cb3c5d92b2f7a67535c784932e6816ba554 from index: no such id: '07784e9c56c0fb3fd964278f0cb15aadc16d80ff34ba3cc44093c91284e2d8fe'"
Sep 17 21:08:13.052639 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:13.052612    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b61317-predictor-6475c545c4-dk2kz_b5c7b2df-6d85-45f5-9190-d81cf88321e6/storage-initializer/1.log"
Sep 17 21:08:16.841806 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:16.841772    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 21:08:16.842283 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:16.842261    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:08:19.132971 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:19.132933    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz"]
Sep 17 21:08:19.263219 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:19.263191    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b61317-predictor-6475c545c4-dk2kz_b5c7b2df-6d85-45f5-9190-d81cf88321e6/storage-initializer/1.log"
Sep 17 21:08:19.266992 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:19.266976    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz"
Sep 17 21:08:19.322538 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:19.322506    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5c7b2df-6d85-45f5-9190-d81cf88321e6-cabundle-cert\") pod \"b5c7b2df-6d85-45f5-9190-d81cf88321e6\" (UID: \"b5c7b2df-6d85-45f5-9190-d81cf88321e6\") "
Sep 17 21:08:19.322694 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:19.322551    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5c7b2df-6d85-45f5-9190-d81cf88321e6-kserve-provision-location\") pod \"b5c7b2df-6d85-45f5-9190-d81cf88321e6\" (UID: \"b5c7b2df-6d85-45f5-9190-d81cf88321e6\") "
Sep 17 21:08:19.322694 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:19.322582    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5c7b2df-6d85-45f5-9190-d81cf88321e6-proxy-tls\") pod \"b5c7b2df-6d85-45f5-9190-d81cf88321e6\" (UID: \"b5c7b2df-6d85-45f5-9190-d81cf88321e6\") "
Sep 17 21:08:19.322694 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:19.322610    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8vgs\" (UniqueName: \"kubernetes.io/projected/b5c7b2df-6d85-45f5-9190-d81cf88321e6-kube-api-access-r8vgs\") pod \"b5c7b2df-6d85-45f5-9190-d81cf88321e6\" (UID: \"b5c7b2df-6d85-45f5-9190-d81cf88321e6\") "
Sep 17 21:08:19.322694 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:19.322676    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5c7b2df-6d85-45f5-9190-d81cf88321e6-kube-rbac-proxy-sar-config\") pod \"b5c7b2df-6d85-45f5-9190-d81cf88321e6\" (UID: \"b5c7b2df-6d85-45f5-9190-d81cf88321e6\") "
Sep 17 21:08:19.322899 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:19.322873    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5c7b2df-6d85-45f5-9190-d81cf88321e6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b5c7b2df-6d85-45f5-9190-d81cf88321e6" (UID: "b5c7b2df-6d85-45f5-9190-d81cf88321e6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:08:19.322939 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:19.322920    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5c7b2df-6d85-45f5-9190-d81cf88321e6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b5c7b2df-6d85-45f5-9190-d81cf88321e6" (UID: "b5c7b2df-6d85-45f5-9190-d81cf88321e6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:08:19.323072 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:19.323047    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5c7b2df-6d85-45f5-9190-d81cf88321e6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b5c7b2df-6d85-45f5-9190-d81cf88321e6" (UID: "b5c7b2df-6d85-45f5-9190-d81cf88321e6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:08:19.324711 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:19.324685    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5c7b2df-6d85-45f5-9190-d81cf88321e6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b5c7b2df-6d85-45f5-9190-d81cf88321e6" (UID: "b5c7b2df-6d85-45f5-9190-d81cf88321e6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:08:19.324806 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:19.324750    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5c7b2df-6d85-45f5-9190-d81cf88321e6-kube-api-access-r8vgs" (OuterVolumeSpecName: "kube-api-access-r8vgs") pod "b5c7b2df-6d85-45f5-9190-d81cf88321e6" (UID: "b5c7b2df-6d85-45f5-9190-d81cf88321e6"). InnerVolumeSpecName "kube-api-access-r8vgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:08:19.424120 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:19.424036    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5c7b2df-6d85-45f5-9190-d81cf88321e6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:08:19.424120 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:19.424060    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5c7b2df-6d85-45f5-9190-d81cf88321e6-cabundle-cert\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:08:19.424120 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:19.424069    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5c7b2df-6d85-45f5-9190-d81cf88321e6-kserve-provision-location\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:08:19.424120 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:19.424079    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5c7b2df-6d85-45f5-9190-d81cf88321e6-proxy-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:08:19.424120 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:19.424087    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8vgs\" (UniqueName: \"kubernetes.io/projected/b5c7b2df-6d85-45f5-9190-d81cf88321e6-kube-api-access-r8vgs\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:08:20.081907 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:20.081881    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b61317-predictor-6475c545c4-dk2kz_b5c7b2df-6d85-45f5-9190-d81cf88321e6/storage-initializer/1.log"
Sep 17 21:08:20.082074 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:20.081972    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz"
Sep 17 21:08:20.082074 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:20.081981    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz" event={"ID":"b5c7b2df-6d85-45f5-9190-d81cf88321e6","Type":"ContainerDied","Data":"f9826a9001eb57d8091746fc211f9cb3c5d92b2f7a67535c784932e6816ba554"}
Sep 17 21:08:20.082074 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:20.082049    2566 scope.go:117] "RemoveContainer" containerID="d848f5e44bc5d359ee07bde3d963947bf3804d9766bc3cdb8ba6633f16c7b6b2"
Sep 17 21:08:20.118727 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:20.118705    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz"]
Sep 17 21:08:20.124437 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:20.124413    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b61317-predictor-6475c545c4-dk2kz"]
Sep 17 21:08:20.460353 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:20.460281    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5c7b2df-6d85-45f5-9190-d81cf88321e6" path="/var/lib/kubelet/pods/b5c7b2df-6d85-45f5-9190-d81cf88321e6/volumes"
Sep 17 21:08:24.355128 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:24.355095    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx"]
Sep 17 21:08:24.355533 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:24.355432    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5c7b2df-6d85-45f5-9190-d81cf88321e6" containerName="storage-initializer"
Sep 17 21:08:24.355533 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:24.355443    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5c7b2df-6d85-45f5-9190-d81cf88321e6" containerName="storage-initializer"
Sep 17 21:08:24.355533 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:24.355509    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5c7b2df-6d85-45f5-9190-d81cf88321e6" containerName="storage-initializer"
Sep 17 21:08:24.355653 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:24.355572    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5c7b2df-6d85-45f5-9190-d81cf88321e6" containerName="storage-initializer"
Sep 17 21:08:24.355653 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:24.355578    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5c7b2df-6d85-45f5-9190-d81cf88321e6" containerName="storage-initializer"
Sep 17 21:08:24.355653 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:24.355637    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5c7b2df-6d85-45f5-9190-d81cf88321e6" containerName="storage-initializer"
Sep 17 21:08:24.360758 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:24.360739    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx"
Sep 17 21:08:24.363799 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:24.363776    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-042be1-kube-rbac-proxy-sar-config\""
Sep 17 21:08:24.363925 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:24.363865    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-042be1-predictor-serving-cert\""
Sep 17 21:08:24.363925 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:24.363869    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-042be1-dockercfg-424dg\""
Sep 17 21:08:24.363925 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:24.363894    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-042be1\""
Sep 17 21:08:24.377398 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:24.377369    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx"]
Sep 17 21:08:24.464488 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:24.464466    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6-proxy-tls\") pod \"isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx\" (UID: \"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6\") " pod="kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx"
Sep 17 21:08:24.464617 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:24.464508    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6-kserve-provision-location\") pod \"isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx\" (UID: \"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6\") " pod="kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx"
Sep 17 21:08:24.464617 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:24.464526    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx\" (UID: \"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6\") " pod="kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx"
Sep 17 21:08:24.464617 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:24.464545    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn9qp\" (UniqueName: \"kubernetes.io/projected/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6-kube-api-access-gn9qp\") pod \"isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx\" (UID: \"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6\") " pod="kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx"
Sep 17 21:08:24.464765 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:24.464623    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6-cabundle-cert\") pod \"isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx\" (UID: \"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6\") " pod="kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx"
Sep 17 21:08:24.565027 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:24.564996    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6-cabundle-cert\") pod \"isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx\" (UID: \"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6\") " pod="kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx"
Sep 17 21:08:24.565164 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:24.565068    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6-proxy-tls\") pod \"isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx\" (UID: \"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6\") " pod="kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx"
Sep 17 21:08:24.565164 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:24.565134    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6-kserve-provision-location\") pod \"isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx\" (UID: \"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6\") " pod="kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx"
Sep 17 21:08:24.565164 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:24.565158    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx\" (UID: \"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6\") " pod="kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx"
Sep 17 21:08:24.565364 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:24.565186    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn9qp\" (UniqueName: \"kubernetes.io/projected/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6-kube-api-access-gn9qp\") pod \"isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx\" (UID: \"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6\") " pod="kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx"
Sep 17 21:08:24.565569 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:24.565548    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6-kserve-provision-location\") pod \"isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx\" (UID: \"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6\") " pod="kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx"
Sep 17 21:08:24.565830 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:24.565814    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6-cabundle-cert\") pod \"isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx\" (UID: \"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6\") " pod="kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx"
Sep 17 21:08:24.565882 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:24.565862    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx\" (UID: \"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6\") " pod="kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx"
Sep 17 21:08:24.567598 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:24.567582    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6-proxy-tls\") pod \"isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx\" (UID: \"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6\") " pod="kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx"
Sep 17 21:08:24.582156 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:24.582137    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn9qp\" (UniqueName: \"kubernetes.io/projected/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6-kube-api-access-gn9qp\") pod \"isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx\" (UID: \"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6\") " pod="kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx"
Sep 17 21:08:24.670625 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:24.670556    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx"
Sep 17 21:08:24.939885 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:24.939863    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx"]
Sep 17 21:08:24.942340 ip-10-0-138-140 kubenswrapper[2566]: W0917 21:08:24.942311    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f6f4469_dbdb_45cb_adfc_d10762ef5ec6.slice/crio-51736c19b782553a696f4fc182916ea7e98d0126b2f08e4a733d2aac69df6769 WatchSource:0}: Error finding container 51736c19b782553a696f4fc182916ea7e98d0126b2f08e4a733d2aac69df6769: Status 404 returned error can't find the container with id 51736c19b782553a696f4fc182916ea7e98d0126b2f08e4a733d2aac69df6769
Sep 17 21:08:25.107817 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:25.107776    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx" event={"ID":"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6","Type":"ContainerStarted","Data":"e7f7dba18d3107534ecdc2c6eb358136d77959c389fb8f387508cfd42fae53c5"}
Sep 17 21:08:25.108010 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:25.107860    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx" event={"ID":"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6","Type":"ContainerStarted","Data":"51736c19b782553a696f4fc182916ea7e98d0126b2f08e4a733d2aac69df6769"}
Sep 17 21:08:26.841940 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:26.841905    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 21:08:26.842463 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:26.842416    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:08:29.125382 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:29.125302    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx_8f6f4469-dbdb-45cb-adfc-d10762ef5ec6/storage-initializer/0.log"
Sep 17 21:08:29.125382 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:29.125344    2566 generic.go:356] "Generic (PLEG): container finished" podID="8f6f4469-dbdb-45cb-adfc-d10762ef5ec6" containerID="e7f7dba18d3107534ecdc2c6eb358136d77959c389fb8f387508cfd42fae53c5" exitCode=1
Sep 17 21:08:29.125761 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:29.125418    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx" event={"ID":"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6","Type":"ContainerDied","Data":"e7f7dba18d3107534ecdc2c6eb358136d77959c389fb8f387508cfd42fae53c5"}
Sep 17 21:08:30.131194 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:30.131166    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx_8f6f4469-dbdb-45cb-adfc-d10762ef5ec6/storage-initializer/0.log"
Sep 17 21:08:30.131571 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:30.131237    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx" event={"ID":"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6","Type":"ContainerStarted","Data":"fdbea66309d7caf212288ec99a082af395bb3a28f44b744b9cdc73ba7fa36168"}
Sep 17 21:08:34.309167 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:34.309091    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx"]
Sep 17 21:08:34.309628 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:34.309308    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx" podUID="8f6f4469-dbdb-45cb-adfc-d10762ef5ec6" containerName="storage-initializer" containerID="cri-o://fdbea66309d7caf212288ec99a082af395bb3a28f44b744b9cdc73ba7fa36168" gracePeriod=30
Sep 17 21:08:35.664837 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:35.664813    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx_8f6f4469-dbdb-45cb-adfc-d10762ef5ec6/storage-initializer/1.log"
Sep 17 21:08:35.665246 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:35.665199    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx_8f6f4469-dbdb-45cb-adfc-d10762ef5ec6/storage-initializer/0.log"
Sep 17 21:08:35.667639 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:35.667623    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx"
Sep 17 21:08:35.758049 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:35.758027    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6-cabundle-cert\") pod \"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6\" (UID: \"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6\") "
Sep 17 21:08:35.758154 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:35.758099    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6-kserve-provision-location\") pod \"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6\" (UID: \"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6\") "
Sep 17 21:08:35.758154 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:35.758134    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6-proxy-tls\") pod \"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6\" (UID: \"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6\") "
Sep 17 21:08:35.758268 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:35.758191    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6-kube-rbac-proxy-sar-config\") pod \"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6\" (UID: \"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6\") "
Sep 17 21:08:35.758268 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:35.758248    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gn9qp\" (UniqueName: \"kubernetes.io/projected/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6-kube-api-access-gn9qp\") pod \"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6\" (UID: \"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6\") "
Sep 17 21:08:35.758378 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:35.758327    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8f6f4469-dbdb-45cb-adfc-d10762ef5ec6" (UID: "8f6f4469-dbdb-45cb-adfc-d10762ef5ec6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:08:35.758522 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:35.758496    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8f6f4469-dbdb-45cb-adfc-d10762ef5ec6" (UID: "8f6f4469-dbdb-45cb-adfc-d10762ef5ec6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:08:35.758522 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:35.758502    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6-kserve-provision-location\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:08:35.758634 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:35.758596    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8f6f4469-dbdb-45cb-adfc-d10762ef5ec6" (UID: "8f6f4469-dbdb-45cb-adfc-d10762ef5ec6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:08:35.760273 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:35.760254    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8f6f4469-dbdb-45cb-adfc-d10762ef5ec6" (UID: "8f6f4469-dbdb-45cb-adfc-d10762ef5ec6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:08:35.760328 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:35.760311    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6-kube-api-access-gn9qp" (OuterVolumeSpecName: "kube-api-access-gn9qp") pod "8f6f4469-dbdb-45cb-adfc-d10762ef5ec6" (UID: "8f6f4469-dbdb-45cb-adfc-d10762ef5ec6"). InnerVolumeSpecName "kube-api-access-gn9qp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:08:35.859741 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:35.859706    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6-cabundle-cert\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:08:35.859741 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:35.859734    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6-proxy-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:08:35.859741 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:35.859743    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:08:35.859924 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:35.859751    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn9qp\" (UniqueName: \"kubernetes.io/projected/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6-kube-api-access-gn9qp\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:08:36.155580 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:36.155487    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx_8f6f4469-dbdb-45cb-adfc-d10762ef5ec6/storage-initializer/1.log"
Sep 17 21:08:36.155887 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:36.155867    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx_8f6f4469-dbdb-45cb-adfc-d10762ef5ec6/storage-initializer/0.log"
Sep 17 21:08:36.155992 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:36.155908    2566 generic.go:356] "Generic (PLEG): container finished" podID="8f6f4469-dbdb-45cb-adfc-d10762ef5ec6" containerID="fdbea66309d7caf212288ec99a082af395bb3a28f44b744b9cdc73ba7fa36168" exitCode=1
Sep 17 21:08:36.155992 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:36.155958    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx"
Sep 17 21:08:36.155992 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:36.155974    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx" event={"ID":"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6","Type":"ContainerDied","Data":"fdbea66309d7caf212288ec99a082af395bb3a28f44b744b9cdc73ba7fa36168"}
Sep 17 21:08:36.156150 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:36.156070    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx" event={"ID":"8f6f4469-dbdb-45cb-adfc-d10762ef5ec6","Type":"ContainerDied","Data":"51736c19b782553a696f4fc182916ea7e98d0126b2f08e4a733d2aac69df6769"}
Sep 17 21:08:36.156150 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:36.156113    2566 scope.go:117] "RemoveContainer" containerID="fdbea66309d7caf212288ec99a082af395bb3a28f44b744b9cdc73ba7fa36168"
Sep 17 21:08:36.165099 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:36.165081    2566 scope.go:117] "RemoveContainer" containerID="e7f7dba18d3107534ecdc2c6eb358136d77959c389fb8f387508cfd42fae53c5"
Sep 17 21:08:36.172168 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:36.172148    2566 scope.go:117] "RemoveContainer" containerID="fdbea66309d7caf212288ec99a082af395bb3a28f44b744b9cdc73ba7fa36168"
Sep 17 21:08:36.172427 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:08:36.172407    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdbea66309d7caf212288ec99a082af395bb3a28f44b744b9cdc73ba7fa36168\": container with ID starting with fdbea66309d7caf212288ec99a082af395bb3a28f44b744b9cdc73ba7fa36168 not found: ID does not exist" containerID="fdbea66309d7caf212288ec99a082af395bb3a28f44b744b9cdc73ba7fa36168"
Sep 17 21:08:36.172473 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:36.172436    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdbea66309d7caf212288ec99a082af395bb3a28f44b744b9cdc73ba7fa36168"} err="failed to get container status \"fdbea66309d7caf212288ec99a082af395bb3a28f44b744b9cdc73ba7fa36168\": rpc error: code = NotFound desc = could not find container \"fdbea66309d7caf212288ec99a082af395bb3a28f44b744b9cdc73ba7fa36168\": container with ID starting with fdbea66309d7caf212288ec99a082af395bb3a28f44b744b9cdc73ba7fa36168 not found: ID does not exist"
Sep 17 21:08:36.172473 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:36.172450    2566 scope.go:117] "RemoveContainer" containerID="e7f7dba18d3107534ecdc2c6eb358136d77959c389fb8f387508cfd42fae53c5"
Sep 17 21:08:36.172698 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:08:36.172681    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7f7dba18d3107534ecdc2c6eb358136d77959c389fb8f387508cfd42fae53c5\": container with ID starting with e7f7dba18d3107534ecdc2c6eb358136d77959c389fb8f387508cfd42fae53c5 not found: ID does not exist" containerID="e7f7dba18d3107534ecdc2c6eb358136d77959c389fb8f387508cfd42fae53c5"
Sep 17 21:08:36.172738 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:36.172703    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7f7dba18d3107534ecdc2c6eb358136d77959c389fb8f387508cfd42fae53c5"} err="failed to get container status \"e7f7dba18d3107534ecdc2c6eb358136d77959c389fb8f387508cfd42fae53c5\": rpc error: code = NotFound desc = could not find container \"e7f7dba18d3107534ecdc2c6eb358136d77959c389fb8f387508cfd42fae53c5\": container with ID starting with e7f7dba18d3107534ecdc2c6eb358136d77959c389fb8f387508cfd42fae53c5 not found: ID does not exist"
Sep 17 21:08:36.192244 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:36.192221    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx"]
Sep 17 21:08:36.194785 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:36.194767    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-042be1-predictor-55867d7b6d-6dwjx"]
Sep 17 21:08:36.459658 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:36.459574    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f6f4469-dbdb-45cb-adfc-d10762ef5ec6" path="/var/lib/kubelet/pods/8f6f4469-dbdb-45cb-adfc-d10762ef5ec6/volumes"
Sep 17 21:08:36.842933 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:36.842904    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"
Sep 17 21:08:36.843371 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:36.843015    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"
Sep 17 21:08:39.570077 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:39.570043    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z"]
Sep 17 21:08:39.570615 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:39.570349    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f6f4469-dbdb-45cb-adfc-d10762ef5ec6" containerName="storage-initializer"
Sep 17 21:08:39.570615 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:39.570362    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f6f4469-dbdb-45cb-adfc-d10762ef5ec6" containerName="storage-initializer"
Sep 17 21:08:39.570615 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:39.570426    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f6f4469-dbdb-45cb-adfc-d10762ef5ec6" containerName="storage-initializer"
Sep 17 21:08:39.570615 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:39.570456    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f6f4469-dbdb-45cb-adfc-d10762ef5ec6" containerName="storage-initializer"
Sep 17 21:08:39.570615 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:39.570505    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f6f4469-dbdb-45cb-adfc-d10762ef5ec6" containerName="storage-initializer"
Sep 17 21:08:39.570615 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:39.570511    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f6f4469-dbdb-45cb-adfc-d10762ef5ec6" containerName="storage-initializer"
Sep 17 21:08:39.573501 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:39.573481    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z"
Sep 17 21:08:39.576093 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:39.576072    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-a9ce0-kube-rbac-proxy-sar-config\""
Sep 17 21:08:39.576430 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:39.576415    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-a9ce0-predictor-serving-cert\""
Sep 17 21:08:39.587043 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:39.587023    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z"]
Sep 17 21:08:39.692121 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:39.692091    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b674185e-798a-46b1-ae2b-4d7a0c9e9c82-kserve-provision-location\") pod \"raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z\" (UID: \"b674185e-798a-46b1-ae2b-4d7a0c9e9c82\") " pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z"
Sep 17 21:08:39.692278 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:39.692131    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b674185e-798a-46b1-ae2b-4d7a0c9e9c82-cabundle-cert\") pod \"raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z\" (UID: \"b674185e-798a-46b1-ae2b-4d7a0c9e9c82\") " pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z"
Sep 17 21:08:39.692278 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:39.692156    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8ldb\" (UniqueName: \"kubernetes.io/projected/b674185e-798a-46b1-ae2b-4d7a0c9e9c82-kube-api-access-p8ldb\") pod \"raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z\" (UID: \"b674185e-798a-46b1-ae2b-4d7a0c9e9c82\") " pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z"
Sep 17 21:08:39.692278 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:39.692252    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b674185e-798a-46b1-ae2b-4d7a0c9e9c82-proxy-tls\") pod \"raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z\" (UID: \"b674185e-798a-46b1-ae2b-4d7a0c9e9c82\") " pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z"
Sep 17 21:08:39.692417 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:39.692308    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b674185e-798a-46b1-ae2b-4d7a0c9e9c82-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z\" (UID: \"b674185e-798a-46b1-ae2b-4d7a0c9e9c82\") " pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z"
Sep 17 21:08:39.792907 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:39.792856    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b674185e-798a-46b1-ae2b-4d7a0c9e9c82-cabundle-cert\") pod \"raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z\" (UID: \"b674185e-798a-46b1-ae2b-4d7a0c9e9c82\") " pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z"
Sep 17 21:08:39.793081 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:39.792926    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8ldb\" (UniqueName: \"kubernetes.io/projected/b674185e-798a-46b1-ae2b-4d7a0c9e9c82-kube-api-access-p8ldb\") pod \"raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z\" (UID: \"b674185e-798a-46b1-ae2b-4d7a0c9e9c82\") " pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z"
Sep 17 21:08:39.793081 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:39.792980    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b674185e-798a-46b1-ae2b-4d7a0c9e9c82-proxy-tls\") pod \"raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z\" (UID: \"b674185e-798a-46b1-ae2b-4d7a0c9e9c82\") " pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z"
Sep 17 21:08:39.793081 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:39.793052    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b674185e-798a-46b1-ae2b-4d7a0c9e9c82-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z\" (UID: \"b674185e-798a-46b1-ae2b-4d7a0c9e9c82\") " pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z"
Sep 17 21:08:39.793271 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:39.793100    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b674185e-798a-46b1-ae2b-4d7a0c9e9c82-kserve-provision-location\") pod \"raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z\" (UID: \"b674185e-798a-46b1-ae2b-4d7a0c9e9c82\") " pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z"
Sep 17 21:08:39.793703 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:39.793675    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b674185e-798a-46b1-ae2b-4d7a0c9e9c82-kserve-provision-location\") pod \"raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z\" (UID: \"b674185e-798a-46b1-ae2b-4d7a0c9e9c82\") " pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z"
Sep 17 21:08:39.793870 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:39.793848    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b674185e-798a-46b1-ae2b-4d7a0c9e9c82-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z\" (UID: \"b674185e-798a-46b1-ae2b-4d7a0c9e9c82\") " pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z"
Sep 17 21:08:39.793944 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:39.793871    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b674185e-798a-46b1-ae2b-4d7a0c9e9c82-cabundle-cert\") pod \"raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z\" (UID: \"b674185e-798a-46b1-ae2b-4d7a0c9e9c82\") " pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z"
Sep 17 21:08:39.795701 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:39.795679    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b674185e-798a-46b1-ae2b-4d7a0c9e9c82-proxy-tls\") pod \"raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z\" (UID: \"b674185e-798a-46b1-ae2b-4d7a0c9e9c82\") " pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z"
Sep 17 21:08:39.801621 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:39.801602    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8ldb\" (UniqueName: \"kubernetes.io/projected/b674185e-798a-46b1-ae2b-4d7a0c9e9c82-kube-api-access-p8ldb\") pod \"raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z\" (UID: \"b674185e-798a-46b1-ae2b-4d7a0c9e9c82\") " pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z"
Sep 17 21:08:39.883273 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:39.883193    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z"
Sep 17 21:08:40.153438 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:40.153413    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z"]
Sep 17 21:08:40.155294 ip-10-0-138-140 kubenswrapper[2566]: W0917 21:08:40.155272    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb674185e_798a_46b1_ae2b_4d7a0c9e9c82.slice/crio-3ba85bcb5c00db7bdd7feba757bc40ec3da0a6208379a2874af4955a6e40357e WatchSource:0}: Error finding container 3ba85bcb5c00db7bdd7feba757bc40ec3da0a6208379a2874af4955a6e40357e: Status 404 returned error can't find the container with id 3ba85bcb5c00db7bdd7feba757bc40ec3da0a6208379a2874af4955a6e40357e
Sep 17 21:08:40.175385 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:40.175359    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z" event={"ID":"b674185e-798a-46b1-ae2b-4d7a0c9e9c82","Type":"ContainerStarted","Data":"3ba85bcb5c00db7bdd7feba757bc40ec3da0a6208379a2874af4955a6e40357e"}
Sep 17 21:08:41.180341 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:41.180308    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z" event={"ID":"b674185e-798a-46b1-ae2b-4d7a0c9e9c82","Type":"ContainerStarted","Data":"7d92f0a5d7970f57f9c55fe1a217f9d10104885ac07390226ab2c076568fac70"}
Sep 17 21:08:44.192656 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:44.192568    2566 generic.go:356] "Generic (PLEG): container finished" podID="b674185e-798a-46b1-ae2b-4d7a0c9e9c82" containerID="7d92f0a5d7970f57f9c55fe1a217f9d10104885ac07390226ab2c076568fac70" exitCode=0
Sep 17 21:08:44.192656 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:44.192641    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z" event={"ID":"b674185e-798a-46b1-ae2b-4d7a0c9e9c82","Type":"ContainerDied","Data":"7d92f0a5d7970f57f9c55fe1a217f9d10104885ac07390226ab2c076568fac70"}
Sep 17 21:08:45.198498 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:45.198462    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z" event={"ID":"b674185e-798a-46b1-ae2b-4d7a0c9e9c82","Type":"ContainerStarted","Data":"1405ab05ddc2123b7e2477bf79ebdeb2228f73f02b4e75139775e2075622f737"}
Sep 17 21:08:45.198881 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:45.198523    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z" event={"ID":"b674185e-798a-46b1-ae2b-4d7a0c9e9c82","Type":"ContainerStarted","Data":"8cde1b023d398c1ba8dcf337a1f08867457869e2d76bb523fc75e1f4a30f3117"}
Sep 17 21:08:45.198881 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:45.198801    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z"
Sep 17 21:08:45.198960 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:45.198933    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z"
Sep 17 21:08:45.200329 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:45.200301    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z" podUID="b674185e-798a-46b1-ae2b-4d7a0c9e9c82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 21:08:45.219976 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:45.219931    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z" podStartSLOduration=2.581256393 podStartE2EDuration="6.219920038s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.638663645s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:08:45.217904115 +0000 UTC m=+1166.828637343" watchObservedRunningTime="2026-09-17 21:08:45.219920038 +0000 UTC m=+1166.830653267"
Sep 17 21:08:46.202729 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:46.202692    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z" podUID="b674185e-798a-46b1-ae2b-4d7a0c9e9c82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 21:08:49.165227 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:49.165174    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"]
Sep 17 21:08:49.165800 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:49.165447    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="kserve-container" containerID="cri-o://6d55c9765ef22826a335df0f5a984731a79a4febd352f575af945f16d157bc4b" gracePeriod=30
Sep 17 21:08:49.165800 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:49.165484    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="kube-rbac-proxy" containerID="cri-o://0b2774f2aa249db63c52d1ce9a8c6af8b63663893d944e3bd459cb44db0c9759" gracePeriod=30
Sep 17 21:08:49.165800 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:49.165447    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="agent" containerID="cri-o://0ba165e0e0738b6ef8212ef7f0d79a8384e2c80624b40251404536cceb15c1cc" gracePeriod=30
Sep 17 21:08:50.219523 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:50.219488    2566 generic.go:356] "Generic (PLEG): container finished" podID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerID="0b2774f2aa249db63c52d1ce9a8c6af8b63663893d944e3bd459cb44db0c9759" exitCode=2
Sep 17 21:08:50.219913 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:50.219559    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" event={"ID":"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9","Type":"ContainerDied","Data":"0b2774f2aa249db63c52d1ce9a8c6af8b63663893d944e3bd459cb44db0c9759"}
Sep 17 21:08:51.207699 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:51.207673    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z"
Sep 17 21:08:51.208167 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:51.208127    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z" podUID="b674185e-798a-46b1-ae2b-4d7a0c9e9c82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 21:08:51.837788 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:51.837748    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 21:08:54.237461 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:54.237424    2566 generic.go:356] "Generic (PLEG): container finished" podID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerID="6d55c9765ef22826a335df0f5a984731a79a4febd352f575af945f16d157bc4b" exitCode=0
Sep 17 21:08:54.237820 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:54.237496    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" event={"ID":"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9","Type":"ContainerDied","Data":"6d55c9765ef22826a335df0f5a984731a79a4febd352f575af945f16d157bc4b"}
Sep 17 21:08:56.837384 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:56.837344    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 21:08:56.841633 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:56.841606    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 21:08:56.841938 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:08:56.841919    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:09:01.208711 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:01.208669    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z" podUID="b674185e-798a-46b1-ae2b-4d7a0c9e9c82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 21:09:01.837664 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:01.837619    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 21:09:01.837863 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:01.837769    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"
Sep 17 21:09:06.837164 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:06.837125    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 21:09:06.841592 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:06.841564    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 21:09:06.842009 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:06.841986    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:09:11.208385 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:11.208348    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z" podUID="b674185e-798a-46b1-ae2b-4d7a0c9e9c82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 21:09:11.837414 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:11.837373    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 21:09:16.837995 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:16.837957    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 21:09:16.842282 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:16.842250    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 21:09:16.842419 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:16.842400    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"
Sep 17 21:09:16.842683 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:16.842654    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:09:16.842790 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:16.842745    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"
Sep 17 21:09:19.309242 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.309204    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"
Sep 17 21:09:19.341269 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.341240    2566 generic.go:356] "Generic (PLEG): container finished" podID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerID="0ba165e0e0738b6ef8212ef7f0d79a8384e2c80624b40251404536cceb15c1cc" exitCode=0
Sep 17 21:09:19.341406 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.341305    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"
Sep 17 21:09:19.341406 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.341331    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" event={"ID":"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9","Type":"ContainerDied","Data":"0ba165e0e0738b6ef8212ef7f0d79a8384e2c80624b40251404536cceb15c1cc"}
Sep 17 21:09:19.341493 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.341417    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w" event={"ID":"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9","Type":"ContainerDied","Data":"2ff80ef14580fcae01b5bb4b61c13845689844253b33037808fc74fa566a881e"}
Sep 17 21:09:19.341493 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.341468    2566 scope.go:117] "RemoveContainer" containerID="0ba165e0e0738b6ef8212ef7f0d79a8384e2c80624b40251404536cceb15c1cc"
Sep 17 21:09:19.350790 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.350772    2566 scope.go:117] "RemoveContainer" containerID="0b2774f2aa249db63c52d1ce9a8c6af8b63663893d944e3bd459cb44db0c9759"
Sep 17 21:09:19.357918 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.357903    2566 scope.go:117] "RemoveContainer" containerID="6d55c9765ef22826a335df0f5a984731a79a4febd352f575af945f16d157bc4b"
Sep 17 21:09:19.364500 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.364484    2566 scope.go:117] "RemoveContainer" containerID="bd48be16c55a2539b5af500393af0b7e7d7acf89add20f69587b49838c215d0a"
Sep 17 21:09:19.371051 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.371026    2566 scope.go:117] "RemoveContainer" containerID="0ba165e0e0738b6ef8212ef7f0d79a8384e2c80624b40251404536cceb15c1cc"
Sep 17 21:09:19.371347 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:09:19.371323    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ba165e0e0738b6ef8212ef7f0d79a8384e2c80624b40251404536cceb15c1cc\": container with ID starting with 0ba165e0e0738b6ef8212ef7f0d79a8384e2c80624b40251404536cceb15c1cc not found: ID does not exist" containerID="0ba165e0e0738b6ef8212ef7f0d79a8384e2c80624b40251404536cceb15c1cc"
Sep 17 21:09:19.371417 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.371355    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ba165e0e0738b6ef8212ef7f0d79a8384e2c80624b40251404536cceb15c1cc"} err="failed to get container status \"0ba165e0e0738b6ef8212ef7f0d79a8384e2c80624b40251404536cceb15c1cc\": rpc error: code = NotFound desc = could not find container \"0ba165e0e0738b6ef8212ef7f0d79a8384e2c80624b40251404536cceb15c1cc\": container with ID starting with 0ba165e0e0738b6ef8212ef7f0d79a8384e2c80624b40251404536cceb15c1cc not found: ID does not exist"
Sep 17 21:09:19.371417 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.371373    2566 scope.go:117] "RemoveContainer" containerID="0b2774f2aa249db63c52d1ce9a8c6af8b63663893d944e3bd459cb44db0c9759"
Sep 17 21:09:19.371614 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:09:19.371596    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b2774f2aa249db63c52d1ce9a8c6af8b63663893d944e3bd459cb44db0c9759\": container with ID starting with 0b2774f2aa249db63c52d1ce9a8c6af8b63663893d944e3bd459cb44db0c9759 not found: ID does not exist" containerID="0b2774f2aa249db63c52d1ce9a8c6af8b63663893d944e3bd459cb44db0c9759"
Sep 17 21:09:19.371652 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.371620    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b2774f2aa249db63c52d1ce9a8c6af8b63663893d944e3bd459cb44db0c9759"} err="failed to get container status \"0b2774f2aa249db63c52d1ce9a8c6af8b63663893d944e3bd459cb44db0c9759\": rpc error: code = NotFound desc = could not find container \"0b2774f2aa249db63c52d1ce9a8c6af8b63663893d944e3bd459cb44db0c9759\": container with ID starting with 0b2774f2aa249db63c52d1ce9a8c6af8b63663893d944e3bd459cb44db0c9759 not found: ID does not exist"
Sep 17 21:09:19.371652 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.371636    2566 scope.go:117] "RemoveContainer" containerID="6d55c9765ef22826a335df0f5a984731a79a4febd352f575af945f16d157bc4b"
Sep 17 21:09:19.371845 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:09:19.371829    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d55c9765ef22826a335df0f5a984731a79a4febd352f575af945f16d157bc4b\": container with ID starting with 6d55c9765ef22826a335df0f5a984731a79a4febd352f575af945f16d157bc4b not found: ID does not exist" containerID="6d55c9765ef22826a335df0f5a984731a79a4febd352f575af945f16d157bc4b"
Sep 17 21:09:19.371891 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.371846    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d55c9765ef22826a335df0f5a984731a79a4febd352f575af945f16d157bc4b"} err="failed to get container status \"6d55c9765ef22826a335df0f5a984731a79a4febd352f575af945f16d157bc4b\": rpc error: code = NotFound desc = could not find container \"6d55c9765ef22826a335df0f5a984731a79a4febd352f575af945f16d157bc4b\": container with ID starting with 6d55c9765ef22826a335df0f5a984731a79a4febd352f575af945f16d157bc4b not found: ID does not exist"
Sep 17 21:09:19.371891 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.371857    2566 scope.go:117] "RemoveContainer" containerID="bd48be16c55a2539b5af500393af0b7e7d7acf89add20f69587b49838c215d0a"
Sep 17 21:09:19.372052 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:09:19.372037    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd48be16c55a2539b5af500393af0b7e7d7acf89add20f69587b49838c215d0a\": container with ID starting with bd48be16c55a2539b5af500393af0b7e7d7acf89add20f69587b49838c215d0a not found: ID does not exist" containerID="bd48be16c55a2539b5af500393af0b7e7d7acf89add20f69587b49838c215d0a"
Sep 17 21:09:19.372095 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.372054    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd48be16c55a2539b5af500393af0b7e7d7acf89add20f69587b49838c215d0a"} err="failed to get container status \"bd48be16c55a2539b5af500393af0b7e7d7acf89add20f69587b49838c215d0a\": rpc error: code = NotFound desc = could not find container \"bd48be16c55a2539b5af500393af0b7e7d7acf89add20f69587b49838c215d0a\": container with ID starting with bd48be16c55a2539b5af500393af0b7e7d7acf89add20f69587b49838c215d0a not found: ID does not exist"
Sep 17 21:09:19.402367 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.402348    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-cabundle-cert\") pod \"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9\" (UID: \"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9\") "
Sep 17 21:09:19.402479 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.402376    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-proxy-tls\") pod \"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9\" (UID: \"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9\") "
Sep 17 21:09:19.402479 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.402411    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-kube-rbac-proxy-sar-config\") pod \"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9\" (UID: \"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9\") "
Sep 17 21:09:19.402479 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.402427    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkx89\" (UniqueName: \"kubernetes.io/projected/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-kube-api-access-zkx89\") pod \"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9\" (UID: \"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9\") "
Sep 17 21:09:19.402479 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.402475    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-kserve-provision-location\") pod \"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9\" (UID: \"46cdee62-f1b1-4a8b-bd3c-0717fb883fe9\") "
Sep 17 21:09:19.402848 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.402822    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" (UID: "46cdee62-f1b1-4a8b-bd3c-0717fb883fe9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:09:19.402848 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.402832    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" (UID: "46cdee62-f1b1-4a8b-bd3c-0717fb883fe9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:09:19.402971 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.402855    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" (UID: "46cdee62-f1b1-4a8b-bd3c-0717fb883fe9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:09:19.404306 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.404282    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" (UID: "46cdee62-f1b1-4a8b-bd3c-0717fb883fe9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:09:19.404443 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.404427    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-kube-api-access-zkx89" (OuterVolumeSpecName: "kube-api-access-zkx89") pod "46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" (UID: "46cdee62-f1b1-4a8b-bd3c-0717fb883fe9"). InnerVolumeSpecName "kube-api-access-zkx89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:09:19.503471 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.503448    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:09:19.503471 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.503468    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkx89\" (UniqueName: \"kubernetes.io/projected/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-kube-api-access-zkx89\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:09:19.503602 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.503477    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-kserve-provision-location\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:09:19.503602 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.503487    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-cabundle-cert\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:09:19.503602 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.503495    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9-proxy-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:09:19.662866 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.662834    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"]
Sep 17 21:09:19.668769 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:19.668748    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-8d8d6-predictor-78c477c7b7-zd86w"]
Sep 17 21:09:20.444888 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:20.444857    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t2rlv_ed6472a6-7979-43b0-a549-7ca5c5d8fb0f/console-operator/1.log"
Sep 17 21:09:20.446766 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:20.446746    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t2rlv_ed6472a6-7979-43b0-a549-7ca5c5d8fb0f/console-operator/1.log"
Sep 17 21:09:20.450443 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:20.450427    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnqx7_402f55a4-24b6-47c6-b719-9c6a5b473434/ovn-acl-logging/0.log"
Sep 17 21:09:20.452161 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:20.452145    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnqx7_402f55a4-24b6-47c6-b719-9c6a5b473434/ovn-acl-logging/0.log"
Sep 17 21:09:20.459561 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:20.459531    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" path="/var/lib/kubelet/pods/46cdee62-f1b1-4a8b-bd3c-0717fb883fe9/volumes"
Sep 17 21:09:21.208507 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:21.208465    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z" podUID="b674185e-798a-46b1-ae2b-4d7a0c9e9c82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 21:09:31.208776 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:31.208740    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z" podUID="b674185e-798a-46b1-ae2b-4d7a0c9e9c82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 21:09:41.208631 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:41.208591    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z" podUID="b674185e-798a-46b1-ae2b-4d7a0c9e9c82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 21:09:51.208678 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:09:51.208638    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z" podUID="b674185e-798a-46b1-ae2b-4d7a0c9e9c82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 21:10:01.208824 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:01.208791    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z"
Sep 17 21:10:10.093235 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:10.093143    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z"]
Sep 17 21:10:10.093636 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:10.093390    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z" podUID="b674185e-798a-46b1-ae2b-4d7a0c9e9c82" containerName="kserve-container" containerID="cri-o://8cde1b023d398c1ba8dcf337a1f08867457869e2d76bb523fc75e1f4a30f3117" gracePeriod=30
Sep 17 21:10:10.093636 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:10.093430    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z" podUID="b674185e-798a-46b1-ae2b-4d7a0c9e9c82" containerName="kube-rbac-proxy" containerID="cri-o://1405ab05ddc2123b7e2477bf79ebdeb2228f73f02b4e75139775e2075622f737" gracePeriod=30
Sep 17 21:10:10.541171 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:10.541137    2566 generic.go:356] "Generic (PLEG): container finished" podID="b674185e-798a-46b1-ae2b-4d7a0c9e9c82" containerID="1405ab05ddc2123b7e2477bf79ebdeb2228f73f02b4e75139775e2075622f737" exitCode=2
Sep 17 21:10:10.541171 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:10.541173    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z" event={"ID":"b674185e-798a-46b1-ae2b-4d7a0c9e9c82","Type":"ContainerDied","Data":"1405ab05ddc2123b7e2477bf79ebdeb2228f73f02b4e75139775e2075622f737"}
Sep 17 21:10:11.203024 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:11.202978    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z" podUID="b674185e-798a-46b1-ae2b-4d7a0c9e9c82" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 17 21:10:11.208355 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:11.208328    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z" podUID="b674185e-798a-46b1-ae2b-4d7a0c9e9c82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 21:10:14.558464 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:14.558433    2566 generic.go:356] "Generic (PLEG): container finished" podID="b674185e-798a-46b1-ae2b-4d7a0c9e9c82" containerID="8cde1b023d398c1ba8dcf337a1f08867457869e2d76bb523fc75e1f4a30f3117" exitCode=0
Sep 17 21:10:14.558799 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:14.558475    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z" event={"ID":"b674185e-798a-46b1-ae2b-4d7a0c9e9c82","Type":"ContainerDied","Data":"8cde1b023d398c1ba8dcf337a1f08867457869e2d76bb523fc75e1f4a30f3117"}
Sep 17 21:10:14.639831 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:14.639804    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z"
Sep 17 21:10:14.698688 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:14.698660    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b674185e-798a-46b1-ae2b-4d7a0c9e9c82-cabundle-cert\") pod \"b674185e-798a-46b1-ae2b-4d7a0c9e9c82\" (UID: \"b674185e-798a-46b1-ae2b-4d7a0c9e9c82\") "
Sep 17 21:10:14.698688 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:14.698692    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b674185e-798a-46b1-ae2b-4d7a0c9e9c82-proxy-tls\") pod \"b674185e-798a-46b1-ae2b-4d7a0c9e9c82\" (UID: \"b674185e-798a-46b1-ae2b-4d7a0c9e9c82\") "
Sep 17 21:10:14.698891 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:14.698742    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b674185e-798a-46b1-ae2b-4d7a0c9e9c82-kube-rbac-proxy-sar-config\") pod \"b674185e-798a-46b1-ae2b-4d7a0c9e9c82\" (UID: \"b674185e-798a-46b1-ae2b-4d7a0c9e9c82\") "
Sep 17 21:10:14.698891 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:14.698768    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8ldb\" (UniqueName: \"kubernetes.io/projected/b674185e-798a-46b1-ae2b-4d7a0c9e9c82-kube-api-access-p8ldb\") pod \"b674185e-798a-46b1-ae2b-4d7a0c9e9c82\" (UID: \"b674185e-798a-46b1-ae2b-4d7a0c9e9c82\") "
Sep 17 21:10:14.698891 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:14.698786    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b674185e-798a-46b1-ae2b-4d7a0c9e9c82-kserve-provision-location\") pod \"b674185e-798a-46b1-ae2b-4d7a0c9e9c82\" (UID: \"b674185e-798a-46b1-ae2b-4d7a0c9e9c82\") "
Sep 17 21:10:14.699123 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:14.699097    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b674185e-798a-46b1-ae2b-4d7a0c9e9c82-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b674185e-798a-46b1-ae2b-4d7a0c9e9c82" (UID: "b674185e-798a-46b1-ae2b-4d7a0c9e9c82"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:10:14.699228 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:14.699131    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b674185e-798a-46b1-ae2b-4d7a0c9e9c82-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b674185e-798a-46b1-ae2b-4d7a0c9e9c82" (UID: "b674185e-798a-46b1-ae2b-4d7a0c9e9c82"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:10:14.699228 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:14.699144    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b674185e-798a-46b1-ae2b-4d7a0c9e9c82-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b674185e-798a-46b1-ae2b-4d7a0c9e9c82" (UID: "b674185e-798a-46b1-ae2b-4d7a0c9e9c82"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:10:14.700881 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:14.700856    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b674185e-798a-46b1-ae2b-4d7a0c9e9c82-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b674185e-798a-46b1-ae2b-4d7a0c9e9c82" (UID: "b674185e-798a-46b1-ae2b-4d7a0c9e9c82"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:10:14.700881 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:14.700868    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b674185e-798a-46b1-ae2b-4d7a0c9e9c82-kube-api-access-p8ldb" (OuterVolumeSpecName: "kube-api-access-p8ldb") pod "b674185e-798a-46b1-ae2b-4d7a0c9e9c82" (UID: "b674185e-798a-46b1-ae2b-4d7a0c9e9c82"). InnerVolumeSpecName "kube-api-access-p8ldb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:10:14.800030 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:14.799966    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b674185e-798a-46b1-ae2b-4d7a0c9e9c82-cabundle-cert\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:10:14.800030 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:14.799989    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b674185e-798a-46b1-ae2b-4d7a0c9e9c82-proxy-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:10:14.800030 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:14.799998    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b674185e-798a-46b1-ae2b-4d7a0c9e9c82-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:10:14.800030 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:14.800007    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8ldb\" (UniqueName: \"kubernetes.io/projected/b674185e-798a-46b1-ae2b-4d7a0c9e9c82-kube-api-access-p8ldb\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:10:14.800030 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:14.800016    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b674185e-798a-46b1-ae2b-4d7a0c9e9c82-kserve-provision-location\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:10:15.295321 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.295281    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x"]
Sep 17 21:10:15.295673 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.295644    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="storage-initializer"
Sep 17 21:10:15.295760 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.295675    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="storage-initializer"
Sep 17 21:10:15.295760 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.295686    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b674185e-798a-46b1-ae2b-4d7a0c9e9c82" containerName="storage-initializer"
Sep 17 21:10:15.295760 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.295694    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b674185e-798a-46b1-ae2b-4d7a0c9e9c82" containerName="storage-initializer"
Sep 17 21:10:15.295760 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.295708    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b674185e-798a-46b1-ae2b-4d7a0c9e9c82" containerName="kube-rbac-proxy"
Sep 17 21:10:15.295760 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.295718    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b674185e-798a-46b1-ae2b-4d7a0c9e9c82" containerName="kube-rbac-proxy"
Sep 17 21:10:15.295760 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.295733    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="kube-rbac-proxy"
Sep 17 21:10:15.295760 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.295741    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="kube-rbac-proxy"
Sep 17 21:10:15.295760 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.295757    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b674185e-798a-46b1-ae2b-4d7a0c9e9c82" containerName="kserve-container"
Sep 17 21:10:15.296176 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.295765    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b674185e-798a-46b1-ae2b-4d7a0c9e9c82" containerName="kserve-container"
Sep 17 21:10:15.296176 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.295797    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="kserve-container"
Sep 17 21:10:15.296176 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.295806    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="kserve-container"
Sep 17 21:10:15.296176 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.295818    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="agent"
Sep 17 21:10:15.296176 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.295827    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="agent"
Sep 17 21:10:15.296176 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.295898    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="kserve-container"
Sep 17 21:10:15.296176 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.295910    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="kube-rbac-proxy"
Sep 17 21:10:15.296176 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.295918    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b674185e-798a-46b1-ae2b-4d7a0c9e9c82" containerName="kserve-container"
Sep 17 21:10:15.296176 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.295928    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="46cdee62-f1b1-4a8b-bd3c-0717fb883fe9" containerName="agent"
Sep 17 21:10:15.296176 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.295938    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b674185e-798a-46b1-ae2b-4d7a0c9e9c82" containerName="kube-rbac-proxy"
Sep 17 21:10:15.299337 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.299317    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x"
Sep 17 21:10:15.302349 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.302333    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-5d5fc-kube-rbac-proxy-sar-config\""
Sep 17 21:10:15.302445 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.302367    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-5d5fc-predictor-serving-cert\""
Sep 17 21:10:15.309558 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.309538    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x"]
Sep 17 21:10:15.404867 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.404841    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71503f3a-f40a-4276-aac9-d08b2d976d4b-kserve-provision-location\") pod \"raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x\" (UID: \"71503f3a-f40a-4276-aac9-d08b2d976d4b\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x"
Sep 17 21:10:15.404979 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.404876    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71503f3a-f40a-4276-aac9-d08b2d976d4b-proxy-tls\") pod \"raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x\" (UID: \"71503f3a-f40a-4276-aac9-d08b2d976d4b\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x"
Sep 17 21:10:15.404979 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.404947    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71503f3a-f40a-4276-aac9-d08b2d976d4b-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x\" (UID: \"71503f3a-f40a-4276-aac9-d08b2d976d4b\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x"
Sep 17 21:10:15.405050 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.404984    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n9bq\" (UniqueName: \"kubernetes.io/projected/71503f3a-f40a-4276-aac9-d08b2d976d4b-kube-api-access-8n9bq\") pod \"raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x\" (UID: \"71503f3a-f40a-4276-aac9-d08b2d976d4b\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x"
Sep 17 21:10:15.405087 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.405047    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71503f3a-f40a-4276-aac9-d08b2d976d4b-cabundle-cert\") pod \"raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x\" (UID: \"71503f3a-f40a-4276-aac9-d08b2d976d4b\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x"
Sep 17 21:10:15.505507 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.505483    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71503f3a-f40a-4276-aac9-d08b2d976d4b-cabundle-cert\") pod \"raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x\" (UID: \"71503f3a-f40a-4276-aac9-d08b2d976d4b\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x"
Sep 17 21:10:15.505584 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.505513    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71503f3a-f40a-4276-aac9-d08b2d976d4b-kserve-provision-location\") pod \"raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x\" (UID: \"71503f3a-f40a-4276-aac9-d08b2d976d4b\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x"
Sep 17 21:10:15.505584 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.505534    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71503f3a-f40a-4276-aac9-d08b2d976d4b-proxy-tls\") pod \"raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x\" (UID: \"71503f3a-f40a-4276-aac9-d08b2d976d4b\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x"
Sep 17 21:10:15.505584 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.505565    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71503f3a-f40a-4276-aac9-d08b2d976d4b-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x\" (UID: \"71503f3a-f40a-4276-aac9-d08b2d976d4b\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x"
Sep 17 21:10:15.505584 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.505581    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n9bq\" (UniqueName: \"kubernetes.io/projected/71503f3a-f40a-4276-aac9-d08b2d976d4b-kube-api-access-8n9bq\") pod \"raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x\" (UID: \"71503f3a-f40a-4276-aac9-d08b2d976d4b\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x"
Sep 17 21:10:15.505739 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:10:15.505703    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-serving-cert: secret "raw-sklearn-runtime-5d5fc-predictor-serving-cert" not found
Sep 17 21:10:15.505789 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:10:15.505778    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71503f3a-f40a-4276-aac9-d08b2d976d4b-proxy-tls podName:71503f3a-f40a-4276-aac9-d08b2d976d4b nodeName:}" failed. No retries permitted until 2026-09-17 21:10:16.005756964 +0000 UTC m=+1257.616490194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/71503f3a-f40a-4276-aac9-d08b2d976d4b-proxy-tls") pod "raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x" (UID: "71503f3a-f40a-4276-aac9-d08b2d976d4b") : secret "raw-sklearn-runtime-5d5fc-predictor-serving-cert" not found
Sep 17 21:10:15.505904 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.505885    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71503f3a-f40a-4276-aac9-d08b2d976d4b-kserve-provision-location\") pod \"raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x\" (UID: \"71503f3a-f40a-4276-aac9-d08b2d976d4b\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x"
Sep 17 21:10:15.506136 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.506118    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71503f3a-f40a-4276-aac9-d08b2d976d4b-cabundle-cert\") pod \"raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x\" (UID: \"71503f3a-f40a-4276-aac9-d08b2d976d4b\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x"
Sep 17 21:10:15.506186 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.506157    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71503f3a-f40a-4276-aac9-d08b2d976d4b-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x\" (UID: \"71503f3a-f40a-4276-aac9-d08b2d976d4b\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x"
Sep 17 21:10:15.515497 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.515475    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n9bq\" (UniqueName: \"kubernetes.io/projected/71503f3a-f40a-4276-aac9-d08b2d976d4b-kube-api-access-8n9bq\") pod \"raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x\" (UID: \"71503f3a-f40a-4276-aac9-d08b2d976d4b\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x"
Sep 17 21:10:15.564173 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.564125    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z" event={"ID":"b674185e-798a-46b1-ae2b-4d7a0c9e9c82","Type":"ContainerDied","Data":"3ba85bcb5c00db7bdd7feba757bc40ec3da0a6208379a2874af4955a6e40357e"}
Sep 17 21:10:15.564478 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.564177    2566 scope.go:117] "RemoveContainer" containerID="1405ab05ddc2123b7e2477bf79ebdeb2228f73f02b4e75139775e2075622f737"
Sep 17 21:10:15.564478 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.564132    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z"
Sep 17 21:10:15.575264 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.575248    2566 scope.go:117] "RemoveContainer" containerID="8cde1b023d398c1ba8dcf337a1f08867457869e2d76bb523fc75e1f4a30f3117"
Sep 17 21:10:15.582147 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.582131    2566 scope.go:117] "RemoveContainer" containerID="7d92f0a5d7970f57f9c55fe1a217f9d10104885ac07390226ab2c076568fac70"
Sep 17 21:10:15.588816 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.588792    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z"]
Sep 17 21:10:15.592612 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:15.592594    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-a9ce0-predictor-948cdbc56-d5d8z"]
Sep 17 21:10:16.010178 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:16.010147    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71503f3a-f40a-4276-aac9-d08b2d976d4b-proxy-tls\") pod \"raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x\" (UID: \"71503f3a-f40a-4276-aac9-d08b2d976d4b\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x"
Sep 17 21:10:16.012596 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:16.012576    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71503f3a-f40a-4276-aac9-d08b2d976d4b-proxy-tls\") pod \"raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x\" (UID: \"71503f3a-f40a-4276-aac9-d08b2d976d4b\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x"
Sep 17 21:10:16.209349 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:16.209318    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x"
Sep 17 21:10:16.459497 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:16.459465    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b674185e-798a-46b1-ae2b-4d7a0c9e9c82" path="/var/lib/kubelet/pods/b674185e-798a-46b1-ae2b-4d7a0c9e9c82/volumes"
Sep 17 21:10:16.487257 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:16.487235    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x"]
Sep 17 21:10:16.571052 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:16.571021    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x" event={"ID":"71503f3a-f40a-4276-aac9-d08b2d976d4b","Type":"ContainerStarted","Data":"4a9a37054892676bc720565b277ec02f922cc769dddb168cb88eef51a122bdaa"}
Sep 17 21:10:16.571455 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:16.571102    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x" event={"ID":"71503f3a-f40a-4276-aac9-d08b2d976d4b","Type":"ContainerStarted","Data":"9ff8058729a4d262b095cd404d87e2a0876ebb35f5c1f54e1416807b1d8b3ef4"}
Sep 17 21:10:20.588674 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:20.588639    2566 generic.go:356] "Generic (PLEG): container finished" podID="71503f3a-f40a-4276-aac9-d08b2d976d4b" containerID="4a9a37054892676bc720565b277ec02f922cc769dddb168cb88eef51a122bdaa" exitCode=0
Sep 17 21:10:20.589043 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:20.588680    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x" event={"ID":"71503f3a-f40a-4276-aac9-d08b2d976d4b","Type":"ContainerDied","Data":"4a9a37054892676bc720565b277ec02f922cc769dddb168cb88eef51a122bdaa"}
Sep 17 21:10:21.600356 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:21.600323    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x" event={"ID":"71503f3a-f40a-4276-aac9-d08b2d976d4b","Type":"ContainerStarted","Data":"39e759d710efd99f8222d4a373353ec384af60386d8bd33a558c2dca4c00cd87"}
Sep 17 21:10:21.600710 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:21.600413    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x" event={"ID":"71503f3a-f40a-4276-aac9-d08b2d976d4b","Type":"ContainerStarted","Data":"3055643395a48e425d4645dc5fe95c0cb048eb06030ea24e4dd219e6f314bd50"}
Sep 17 21:10:21.600753 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:21.600710    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x"
Sep 17 21:10:21.621353 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:21.621312    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x" podStartSLOduration=3.002566167 podStartE2EDuration="6.621299696s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.618733529s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:10:21.620075951 +0000 UTC m=+1263.230809225" watchObservedRunningTime="2026-09-17 21:10:21.621299696 +0000 UTC m=+1263.232032926"
Sep 17 21:10:22.604338 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:22.604307    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x"
Sep 17 21:10:22.605545 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:22.605514    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x" podUID="71503f3a-f40a-4276-aac9-d08b2d976d4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 21:10:23.608382 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:23.608346    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x" podUID="71503f3a-f40a-4276-aac9-d08b2d976d4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 21:10:28.613495 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:28.613469    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x"
Sep 17 21:10:28.613988 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:28.613964    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x" podUID="71503f3a-f40a-4276-aac9-d08b2d976d4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 21:10:38.614707 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:38.614666    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x" podUID="71503f3a-f40a-4276-aac9-d08b2d976d4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 21:10:48.614509 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:48.614470    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x" podUID="71503f3a-f40a-4276-aac9-d08b2d976d4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 21:10:58.614398 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:10:58.614350    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x" podUID="71503f3a-f40a-4276-aac9-d08b2d976d4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 21:11:08.614652 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:08.614615    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x" podUID="71503f3a-f40a-4276-aac9-d08b2d976d4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 21:11:18.614128 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:18.614091    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x" podUID="71503f3a-f40a-4276-aac9-d08b2d976d4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 21:11:28.613910 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:28.613872    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x" podUID="71503f3a-f40a-4276-aac9-d08b2d976d4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 21:11:38.615406 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:38.615331    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x"
Sep 17 21:11:45.716958 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:45.716928    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7"]
Sep 17 21:11:45.720197 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:45.720180    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7"
Sep 17 21:11:45.723586 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:45.723569    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 21:11:45.724190 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:45.724175    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 21:11:45.756161 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:45.756138    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7"]
Sep 17 21:11:45.784784 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:45.784762    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b7d2fc8-6813-41ec-9b52-bae5676260de-proxy-tls\") pod \"isvc-canary-create-predictor-7c7f45c769-tq6h7\" (UID: \"1b7d2fc8-6813-41ec-9b52-bae5676260de\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7"
Sep 17 21:11:45.784888 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:45.784789    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh8xw\" (UniqueName: \"kubernetes.io/projected/1b7d2fc8-6813-41ec-9b52-bae5676260de-kube-api-access-kh8xw\") pod \"isvc-canary-create-predictor-7c7f45c769-tq6h7\" (UID: \"1b7d2fc8-6813-41ec-9b52-bae5676260de\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7"
Sep 17 21:11:45.784888 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:45.784817    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b7d2fc8-6813-41ec-9b52-bae5676260de-cabundle-cert\") pod \"isvc-canary-create-predictor-7c7f45c769-tq6h7\" (UID: \"1b7d2fc8-6813-41ec-9b52-bae5676260de\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7"
Sep 17 21:11:45.785009 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:45.784908    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b7d2fc8-6813-41ec-9b52-bae5676260de-kserve-provision-location\") pod \"isvc-canary-create-predictor-7c7f45c769-tq6h7\" (UID: \"1b7d2fc8-6813-41ec-9b52-bae5676260de\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7"
Sep 17 21:11:45.785009 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:45.784954    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b7d2fc8-6813-41ec-9b52-bae5676260de-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7c7f45c769-tq6h7\" (UID: \"1b7d2fc8-6813-41ec-9b52-bae5676260de\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7"
Sep 17 21:11:45.885633 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:45.885604    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b7d2fc8-6813-41ec-9b52-bae5676260de-kserve-provision-location\") pod \"isvc-canary-create-predictor-7c7f45c769-tq6h7\" (UID: \"1b7d2fc8-6813-41ec-9b52-bae5676260de\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7"
Sep 17 21:11:45.885794 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:45.885639    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b7d2fc8-6813-41ec-9b52-bae5676260de-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7c7f45c769-tq6h7\" (UID: \"1b7d2fc8-6813-41ec-9b52-bae5676260de\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7"
Sep 17 21:11:45.885794 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:45.885691    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b7d2fc8-6813-41ec-9b52-bae5676260de-proxy-tls\") pod \"isvc-canary-create-predictor-7c7f45c769-tq6h7\" (UID: \"1b7d2fc8-6813-41ec-9b52-bae5676260de\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7"
Sep 17 21:11:45.885794 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:45.885708    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh8xw\" (UniqueName: \"kubernetes.io/projected/1b7d2fc8-6813-41ec-9b52-bae5676260de-kube-api-access-kh8xw\") pod \"isvc-canary-create-predictor-7c7f45c769-tq6h7\" (UID: \"1b7d2fc8-6813-41ec-9b52-bae5676260de\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7"
Sep 17 21:11:45.885794 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:45.885743    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b7d2fc8-6813-41ec-9b52-bae5676260de-cabundle-cert\") pod \"isvc-canary-create-predictor-7c7f45c769-tq6h7\" (UID: \"1b7d2fc8-6813-41ec-9b52-bae5676260de\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7"
Sep 17 21:11:45.886063 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:45.886040    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b7d2fc8-6813-41ec-9b52-bae5676260de-kserve-provision-location\") pod \"isvc-canary-create-predictor-7c7f45c769-tq6h7\" (UID: \"1b7d2fc8-6813-41ec-9b52-bae5676260de\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7"
Sep 17 21:11:45.886452 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:45.886403    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b7d2fc8-6813-41ec-9b52-bae5676260de-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7c7f45c769-tq6h7\" (UID: \"1b7d2fc8-6813-41ec-9b52-bae5676260de\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7"
Sep 17 21:11:45.886558 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:45.886477    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b7d2fc8-6813-41ec-9b52-bae5676260de-cabundle-cert\") pod \"isvc-canary-create-predictor-7c7f45c769-tq6h7\" (UID: \"1b7d2fc8-6813-41ec-9b52-bae5676260de\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7"
Sep 17 21:11:45.888118 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:45.888098    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b7d2fc8-6813-41ec-9b52-bae5676260de-proxy-tls\") pod \"isvc-canary-create-predictor-7c7f45c769-tq6h7\" (UID: \"1b7d2fc8-6813-41ec-9b52-bae5676260de\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7"
Sep 17 21:11:45.915273 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:45.915246    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh8xw\" (UniqueName: \"kubernetes.io/projected/1b7d2fc8-6813-41ec-9b52-bae5676260de-kube-api-access-kh8xw\") pod \"isvc-canary-create-predictor-7c7f45c769-tq6h7\" (UID: \"1b7d2fc8-6813-41ec-9b52-bae5676260de\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7"
Sep 17 21:11:46.030370 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:46.030303    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7"
Sep 17 21:11:46.110768 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:46.110730    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x"]
Sep 17 21:11:46.111062 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:46.111032    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x" podUID="71503f3a-f40a-4276-aac9-d08b2d976d4b" containerName="kserve-container" containerID="cri-o://3055643395a48e425d4645dc5fe95c0cb048eb06030ea24e4dd219e6f314bd50" gracePeriod=30
Sep 17 21:11:46.111114 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:46.111072    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x" podUID="71503f3a-f40a-4276-aac9-d08b2d976d4b" containerName="kube-rbac-proxy" containerID="cri-o://39e759d710efd99f8222d4a373353ec384af60386d8bd33a558c2dca4c00cd87" gracePeriod=30
Sep 17 21:11:46.308310 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:46.308242    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7"]
Sep 17 21:11:46.310327 ip-10-0-138-140 kubenswrapper[2566]: W0917 21:11:46.310293    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b7d2fc8_6813_41ec_9b52_bae5676260de.slice/crio-eed1f7805c0a7d888d52309ee9040ef721da85bf3cc2371937f8be41578b6d96 WatchSource:0}: Error finding container eed1f7805c0a7d888d52309ee9040ef721da85bf3cc2371937f8be41578b6d96: Status 404 returned error can't find the container with id eed1f7805c0a7d888d52309ee9040ef721da85bf3cc2371937f8be41578b6d96
Sep 17 21:11:46.939802 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:46.939767    2566 generic.go:356] "Generic (PLEG): container finished" podID="71503f3a-f40a-4276-aac9-d08b2d976d4b" containerID="39e759d710efd99f8222d4a373353ec384af60386d8bd33a558c2dca4c00cd87" exitCode=2
Sep 17 21:11:46.940310 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:46.939832    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x" event={"ID":"71503f3a-f40a-4276-aac9-d08b2d976d4b","Type":"ContainerDied","Data":"39e759d710efd99f8222d4a373353ec384af60386d8bd33a558c2dca4c00cd87"}
Sep 17 21:11:46.941362 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:46.941336    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7" event={"ID":"1b7d2fc8-6813-41ec-9b52-bae5676260de","Type":"ContainerStarted","Data":"eeecb0b3d8e1b46d4371d800f867c17fe500ec97b454865d51e0430ba470486a"}
Sep 17 21:11:46.941485 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:46.941385    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7" event={"ID":"1b7d2fc8-6813-41ec-9b52-bae5676260de","Type":"ContainerStarted","Data":"eed1f7805c0a7d888d52309ee9040ef721da85bf3cc2371937f8be41578b6d96"}
Sep 17 21:11:48.608792 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:48.608755    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x" podUID="71503f3a-f40a-4276-aac9-d08b2d976d4b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 17 21:11:48.614176 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:48.614151    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x" podUID="71503f3a-f40a-4276-aac9-d08b2d976d4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 21:11:50.853190 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:50.853162    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x"
Sep 17 21:11:50.925566 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:50.925533    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71503f3a-f40a-4276-aac9-d08b2d976d4b-cabundle-cert\") pod \"71503f3a-f40a-4276-aac9-d08b2d976d4b\" (UID: \"71503f3a-f40a-4276-aac9-d08b2d976d4b\") "
Sep 17 21:11:50.925700 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:50.925576    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71503f3a-f40a-4276-aac9-d08b2d976d4b-proxy-tls\") pod \"71503f3a-f40a-4276-aac9-d08b2d976d4b\" (UID: \"71503f3a-f40a-4276-aac9-d08b2d976d4b\") "
Sep 17 21:11:50.925700 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:50.925629    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8n9bq\" (UniqueName: \"kubernetes.io/projected/71503f3a-f40a-4276-aac9-d08b2d976d4b-kube-api-access-8n9bq\") pod \"71503f3a-f40a-4276-aac9-d08b2d976d4b\" (UID: \"71503f3a-f40a-4276-aac9-d08b2d976d4b\") "
Sep 17 21:11:50.925700 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:50.925673    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71503f3a-f40a-4276-aac9-d08b2d976d4b-kube-rbac-proxy-sar-config\") pod \"71503f3a-f40a-4276-aac9-d08b2d976d4b\" (UID: \"71503f3a-f40a-4276-aac9-d08b2d976d4b\") "
Sep 17 21:11:50.925908 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:50.925702    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71503f3a-f40a-4276-aac9-d08b2d976d4b-kserve-provision-location\") pod \"71503f3a-f40a-4276-aac9-d08b2d976d4b\" (UID: \"71503f3a-f40a-4276-aac9-d08b2d976d4b\") "
Sep 17 21:11:50.926031 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:50.925912    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71503f3a-f40a-4276-aac9-d08b2d976d4b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "71503f3a-f40a-4276-aac9-d08b2d976d4b" (UID: "71503f3a-f40a-4276-aac9-d08b2d976d4b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:11:50.926072 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:50.926014    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71503f3a-f40a-4276-aac9-d08b2d976d4b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "71503f3a-f40a-4276-aac9-d08b2d976d4b" (UID: "71503f3a-f40a-4276-aac9-d08b2d976d4b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:11:50.926072 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:50.926045    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71503f3a-f40a-4276-aac9-d08b2d976d4b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "71503f3a-f40a-4276-aac9-d08b2d976d4b" (UID: "71503f3a-f40a-4276-aac9-d08b2d976d4b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:11:50.928176 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:50.928159    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71503f3a-f40a-4276-aac9-d08b2d976d4b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "71503f3a-f40a-4276-aac9-d08b2d976d4b" (UID: "71503f3a-f40a-4276-aac9-d08b2d976d4b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:11:50.928239 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:50.928179    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71503f3a-f40a-4276-aac9-d08b2d976d4b-kube-api-access-8n9bq" (OuterVolumeSpecName: "kube-api-access-8n9bq") pod "71503f3a-f40a-4276-aac9-d08b2d976d4b" (UID: "71503f3a-f40a-4276-aac9-d08b2d976d4b"). InnerVolumeSpecName "kube-api-access-8n9bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:11:50.958425 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:50.958362    2566 generic.go:356] "Generic (PLEG): container finished" podID="71503f3a-f40a-4276-aac9-d08b2d976d4b" containerID="3055643395a48e425d4645dc5fe95c0cb048eb06030ea24e4dd219e6f314bd50" exitCode=0
Sep 17 21:11:50.958523 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:50.958432    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x"
Sep 17 21:11:50.958523 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:50.958438    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x" event={"ID":"71503f3a-f40a-4276-aac9-d08b2d976d4b","Type":"ContainerDied","Data":"3055643395a48e425d4645dc5fe95c0cb048eb06030ea24e4dd219e6f314bd50"}
Sep 17 21:11:50.958523 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:50.958509    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x" event={"ID":"71503f3a-f40a-4276-aac9-d08b2d976d4b","Type":"ContainerDied","Data":"9ff8058729a4d262b095cd404d87e2a0876ebb35f5c1f54e1416807b1d8b3ef4"}
Sep 17 21:11:50.958654 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:50.958548    2566 scope.go:117] "RemoveContainer" containerID="39e759d710efd99f8222d4a373353ec384af60386d8bd33a558c2dca4c00cd87"
Sep 17 21:11:50.960168 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:50.960148    2566 generic.go:356] "Generic (PLEG): container finished" podID="1b7d2fc8-6813-41ec-9b52-bae5676260de" containerID="eeecb0b3d8e1b46d4371d800f867c17fe500ec97b454865d51e0430ba470486a" exitCode=0
Sep 17 21:11:50.960361 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:50.960185    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7" event={"ID":"1b7d2fc8-6813-41ec-9b52-bae5676260de","Type":"ContainerDied","Data":"eeecb0b3d8e1b46d4371d800f867c17fe500ec97b454865d51e0430ba470486a"}
Sep 17 21:11:50.967518 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:50.967456    2566 scope.go:117] "RemoveContainer" containerID="3055643395a48e425d4645dc5fe95c0cb048eb06030ea24e4dd219e6f314bd50"
Sep 17 21:11:50.974892 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:50.974875    2566 scope.go:117] "RemoveContainer" containerID="4a9a37054892676bc720565b277ec02f922cc769dddb168cb88eef51a122bdaa"
Sep 17 21:11:50.987339 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:50.987323    2566 scope.go:117] "RemoveContainer" containerID="39e759d710efd99f8222d4a373353ec384af60386d8bd33a558c2dca4c00cd87"
Sep 17 21:11:50.987610 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:11:50.987594    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39e759d710efd99f8222d4a373353ec384af60386d8bd33a558c2dca4c00cd87\": container with ID starting with 39e759d710efd99f8222d4a373353ec384af60386d8bd33a558c2dca4c00cd87 not found: ID does not exist" containerID="39e759d710efd99f8222d4a373353ec384af60386d8bd33a558c2dca4c00cd87"
Sep 17 21:11:50.987674 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:50.987628    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39e759d710efd99f8222d4a373353ec384af60386d8bd33a558c2dca4c00cd87"} err="failed to get container status \"39e759d710efd99f8222d4a373353ec384af60386d8bd33a558c2dca4c00cd87\": rpc error: code = NotFound desc = could not find container \"39e759d710efd99f8222d4a373353ec384af60386d8bd33a558c2dca4c00cd87\": container with ID starting with 39e759d710efd99f8222d4a373353ec384af60386d8bd33a558c2dca4c00cd87 not found: ID does not exist"
Sep 17 21:11:50.987674 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:50.987643    2566 scope.go:117] "RemoveContainer" containerID="3055643395a48e425d4645dc5fe95c0cb048eb06030ea24e4dd219e6f314bd50"
Sep 17 21:11:50.987856 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:11:50.987843    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3055643395a48e425d4645dc5fe95c0cb048eb06030ea24e4dd219e6f314bd50\": container with ID starting with 3055643395a48e425d4645dc5fe95c0cb048eb06030ea24e4dd219e6f314bd50 not found: ID does not exist" containerID="3055643395a48e425d4645dc5fe95c0cb048eb06030ea24e4dd219e6f314bd50"
Sep 17 21:11:50.987896 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:50.987859    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3055643395a48e425d4645dc5fe95c0cb048eb06030ea24e4dd219e6f314bd50"} err="failed to get container status \"3055643395a48e425d4645dc5fe95c0cb048eb06030ea24e4dd219e6f314bd50\": rpc error: code = NotFound desc = could not find container \"3055643395a48e425d4645dc5fe95c0cb048eb06030ea24e4dd219e6f314bd50\": container with ID starting with 3055643395a48e425d4645dc5fe95c0cb048eb06030ea24e4dd219e6f314bd50 not found: ID does not exist"
Sep 17 21:11:50.987896 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:50.987871    2566 scope.go:117] "RemoveContainer" containerID="4a9a37054892676bc720565b277ec02f922cc769dddb168cb88eef51a122bdaa"
Sep 17 21:11:50.988055 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:11:50.988042    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a9a37054892676bc720565b277ec02f922cc769dddb168cb88eef51a122bdaa\": container with ID starting with 4a9a37054892676bc720565b277ec02f922cc769dddb168cb88eef51a122bdaa not found: ID does not exist" containerID="4a9a37054892676bc720565b277ec02f922cc769dddb168cb88eef51a122bdaa"
Sep 17 21:11:50.988096 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:50.988056    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a9a37054892676bc720565b277ec02f922cc769dddb168cb88eef51a122bdaa"} err="failed to get container status \"4a9a37054892676bc720565b277ec02f922cc769dddb168cb88eef51a122bdaa\": rpc error: code = NotFound desc = could not find container \"4a9a37054892676bc720565b277ec02f922cc769dddb168cb88eef51a122bdaa\": container with ID starting with 4a9a37054892676bc720565b277ec02f922cc769dddb168cb88eef51a122bdaa not found: ID does not exist"
Sep 17 21:11:50.995518 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:50.995500    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x"]
Sep 17 21:11:50.999419 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:50.999398    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-5d5fc-predictor-5b8446f67b-bfq9x"]
Sep 17 21:11:51.026283 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:51.026266    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71503f3a-f40a-4276-aac9-d08b2d976d4b-cabundle-cert\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:11:51.026360 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:51.026287    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71503f3a-f40a-4276-aac9-d08b2d976d4b-proxy-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:11:51.026360 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:51.026301    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n9bq\" (UniqueName: \"kubernetes.io/projected/71503f3a-f40a-4276-aac9-d08b2d976d4b-kube-api-access-8n9bq\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:11:51.026360 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:51.026315    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71503f3a-f40a-4276-aac9-d08b2d976d4b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:11:51.026360 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:51.026332    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71503f3a-f40a-4276-aac9-d08b2d976d4b-kserve-provision-location\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:11:51.966785 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:51.966755    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7" event={"ID":"1b7d2fc8-6813-41ec-9b52-bae5676260de","Type":"ContainerStarted","Data":"8eee3c2a7e7f242701c7191c3b38807eff52c63dd7a7b0e287dd972053c0513d"}
Sep 17 21:11:51.967153 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:51.966812    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7" event={"ID":"1b7d2fc8-6813-41ec-9b52-bae5676260de","Type":"ContainerStarted","Data":"314b2c471874969e46158e57b32f3f7292ce75f9013041dbd8eed8f6ec58f4d2"}
Sep 17 21:11:51.967153 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:51.966996    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7"
Sep 17 21:11:51.988891 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:51.988841    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7" podStartSLOduration=3.377374178 podStartE2EDuration="6.988822435s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.611448257s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:11:51.986951279 +0000 UTC m=+1353.597684509" watchObservedRunningTime="2026-09-17 21:11:51.988822435 +0000 UTC m=+1353.599555665"
Sep 17 21:11:52.459079 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:52.459005    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71503f3a-f40a-4276-aac9-d08b2d976d4b" path="/var/lib/kubelet/pods/71503f3a-f40a-4276-aac9-d08b2d976d4b/volumes"
Sep 17 21:11:52.970701 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:52.970667    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7"
Sep 17 21:11:52.971976 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:52.971948    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7" podUID="1b7d2fc8-6813-41ec-9b52-bae5676260de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 21:11:53.973693 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:53.973651    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7" podUID="1b7d2fc8-6813-41ec-9b52-bae5676260de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 21:11:58.977916 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:58.977890    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7"
Sep 17 21:11:58.978401 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:11:58.978369    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7" podUID="1b7d2fc8-6813-41ec-9b52-bae5676260de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 21:12:08.978559 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:12:08.978522    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7" podUID="1b7d2fc8-6813-41ec-9b52-bae5676260de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 21:12:18.979071 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:12:18.979031    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7" podUID="1b7d2fc8-6813-41ec-9b52-bae5676260de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 21:12:28.978949 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:12:28.978905    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7" podUID="1b7d2fc8-6813-41ec-9b52-bae5676260de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 21:12:38.978826 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:12:38.978778    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7" podUID="1b7d2fc8-6813-41ec-9b52-bae5676260de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 21:12:48.978373 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:12:48.978326    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7" podUID="1b7d2fc8-6813-41ec-9b52-bae5676260de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 21:12:58.979348 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:12:58.979304    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7" podUID="1b7d2fc8-6813-41ec-9b52-bae5676260de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 21:13:08.978968 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:08.978895    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7"
Sep 17 21:13:20.877960 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:20.877928    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7"]
Sep 17 21:13:20.878349 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:20.878139    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7" podUID="1b7d2fc8-6813-41ec-9b52-bae5676260de" containerName="kserve-container" containerID="cri-o://314b2c471874969e46158e57b32f3f7292ce75f9013041dbd8eed8f6ec58f4d2" gracePeriod=30
Sep 17 21:13:20.878349 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:20.878189    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7" podUID="1b7d2fc8-6813-41ec-9b52-bae5676260de" containerName="kube-rbac-proxy" containerID="cri-o://8eee3c2a7e7f242701c7191c3b38807eff52c63dd7a7b0e287dd972053c0513d" gracePeriod=30
Sep 17 21:13:21.304085 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:21.304054    2566 generic.go:356] "Generic (PLEG): container finished" podID="1b7d2fc8-6813-41ec-9b52-bae5676260de" containerID="8eee3c2a7e7f242701c7191c3b38807eff52c63dd7a7b0e287dd972053c0513d" exitCode=2
Sep 17 21:13:21.304310 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:21.304133    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7" event={"ID":"1b7d2fc8-6813-41ec-9b52-bae5676260de","Type":"ContainerDied","Data":"8eee3c2a7e7f242701c7191c3b38807eff52c63dd7a7b0e287dd972053c0513d"}
Sep 17 21:13:23.974750 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:23.974708    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7" podUID="1b7d2fc8-6813-41ec-9b52-bae5676260de" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 17 21:13:25.515839 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.515815    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7"
Sep 17 21:13:25.632806 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.632730    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b7d2fc8-6813-41ec-9b52-bae5676260de-cabundle-cert\") pod \"1b7d2fc8-6813-41ec-9b52-bae5676260de\" (UID: \"1b7d2fc8-6813-41ec-9b52-bae5676260de\") "
Sep 17 21:13:25.632806 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.632771    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b7d2fc8-6813-41ec-9b52-bae5676260de-proxy-tls\") pod \"1b7d2fc8-6813-41ec-9b52-bae5676260de\" (UID: \"1b7d2fc8-6813-41ec-9b52-bae5676260de\") "
Sep 17 21:13:25.632806 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.632792    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b7d2fc8-6813-41ec-9b52-bae5676260de-kube-rbac-proxy-sar-config\") pod \"1b7d2fc8-6813-41ec-9b52-bae5676260de\" (UID: \"1b7d2fc8-6813-41ec-9b52-bae5676260de\") "
Sep 17 21:13:25.633082 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.632819    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kh8xw\" (UniqueName: \"kubernetes.io/projected/1b7d2fc8-6813-41ec-9b52-bae5676260de-kube-api-access-kh8xw\") pod \"1b7d2fc8-6813-41ec-9b52-bae5676260de\" (UID: \"1b7d2fc8-6813-41ec-9b52-bae5676260de\") "
Sep 17 21:13:25.633082 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.632844    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b7d2fc8-6813-41ec-9b52-bae5676260de-kserve-provision-location\") pod \"1b7d2fc8-6813-41ec-9b52-bae5676260de\" (UID: \"1b7d2fc8-6813-41ec-9b52-bae5676260de\") "
Sep 17 21:13:25.633249 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.633201    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b7d2fc8-6813-41ec-9b52-bae5676260de-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1b7d2fc8-6813-41ec-9b52-bae5676260de" (UID: "1b7d2fc8-6813-41ec-9b52-bae5676260de"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:13:25.633310 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.633239    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b7d2fc8-6813-41ec-9b52-bae5676260de-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1b7d2fc8-6813-41ec-9b52-bae5676260de" (UID: "1b7d2fc8-6813-41ec-9b52-bae5676260de"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:13:25.633310 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.633270    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7d2fc8-6813-41ec-9b52-bae5676260de-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1b7d2fc8-6813-41ec-9b52-bae5676260de" (UID: "1b7d2fc8-6813-41ec-9b52-bae5676260de"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:13:25.634947 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.634920    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b7d2fc8-6813-41ec-9b52-bae5676260de-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1b7d2fc8-6813-41ec-9b52-bae5676260de" (UID: "1b7d2fc8-6813-41ec-9b52-bae5676260de"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:13:25.635054 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.634951    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7d2fc8-6813-41ec-9b52-bae5676260de-kube-api-access-kh8xw" (OuterVolumeSpecName: "kube-api-access-kh8xw") pod "1b7d2fc8-6813-41ec-9b52-bae5676260de" (UID: "1b7d2fc8-6813-41ec-9b52-bae5676260de"). InnerVolumeSpecName "kube-api-access-kh8xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:13:25.734085 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.734056    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b7d2fc8-6813-41ec-9b52-bae5676260de-cabundle-cert\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:13:25.734085 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.734079    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b7d2fc8-6813-41ec-9b52-bae5676260de-proxy-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:13:25.734085 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.734087    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b7d2fc8-6813-41ec-9b52-bae5676260de-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:13:25.734377 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.734096    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh8xw\" (UniqueName: \"kubernetes.io/projected/1b7d2fc8-6813-41ec-9b52-bae5676260de-kube-api-access-kh8xw\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:13:25.734377 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.734104    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b7d2fc8-6813-41ec-9b52-bae5676260de-kserve-provision-location\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:13:25.945132 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.945052    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s"]
Sep 17 21:13:25.945371 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.945357    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b7d2fc8-6813-41ec-9b52-bae5676260de" containerName="storage-initializer"
Sep 17 21:13:25.945415 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.945373    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b7d2fc8-6813-41ec-9b52-bae5676260de" containerName="storage-initializer"
Sep 17 21:13:25.945415 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.945390    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71503f3a-f40a-4276-aac9-d08b2d976d4b" containerName="storage-initializer"
Sep 17 21:13:25.945415 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.945395    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="71503f3a-f40a-4276-aac9-d08b2d976d4b" containerName="storage-initializer"
Sep 17 21:13:25.945415 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.945407    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b7d2fc8-6813-41ec-9b52-bae5676260de" containerName="kube-rbac-proxy"
Sep 17 21:13:25.945415 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.945414    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b7d2fc8-6813-41ec-9b52-bae5676260de" containerName="kube-rbac-proxy"
Sep 17 21:13:25.945564 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.945421    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b7d2fc8-6813-41ec-9b52-bae5676260de" containerName="kserve-container"
Sep 17 21:13:25.945564 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.945426    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b7d2fc8-6813-41ec-9b52-bae5676260de" containerName="kserve-container"
Sep 17 21:13:25.945564 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.945433    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71503f3a-f40a-4276-aac9-d08b2d976d4b" containerName="kube-rbac-proxy"
Sep 17 21:13:25.945564 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.945438    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="71503f3a-f40a-4276-aac9-d08b2d976d4b" containerName="kube-rbac-proxy"
Sep 17 21:13:25.945564 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.945445    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71503f3a-f40a-4276-aac9-d08b2d976d4b" containerName="kserve-container"
Sep 17 21:13:25.945564 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.945450    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="71503f3a-f40a-4276-aac9-d08b2d976d4b" containerName="kserve-container"
Sep 17 21:13:25.945564 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.945491    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="71503f3a-f40a-4276-aac9-d08b2d976d4b" containerName="kserve-container"
Sep 17 21:13:25.945564 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.945499    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b7d2fc8-6813-41ec-9b52-bae5676260de" containerName="kube-rbac-proxy"
Sep 17 21:13:25.945564 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.945508    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b7d2fc8-6813-41ec-9b52-bae5676260de" containerName="kserve-container"
Sep 17 21:13:25.945564 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.945514    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="71503f3a-f40a-4276-aac9-d08b2d976d4b" containerName="kube-rbac-proxy"
Sep 17 21:13:25.948630 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.948615    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s"
Sep 17 21:13:25.951454 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.951432    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 21:13:25.951853 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.951839    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 21:13:25.976035 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:25.976004    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s"]
Sep 17 21:13:26.036455 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.036428    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/762d65aa-f29f-4025-a118-c61ceeac20b9-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s\" (UID: \"762d65aa-f29f-4025-a118-c61ceeac20b9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s"
Sep 17 21:13:26.036570 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.036458    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/762d65aa-f29f-4025-a118-c61ceeac20b9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s\" (UID: \"762d65aa-f29f-4025-a118-c61ceeac20b9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s"
Sep 17 21:13:26.036570 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.036479    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/762d65aa-f29f-4025-a118-c61ceeac20b9-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s\" (UID: \"762d65aa-f29f-4025-a118-c61ceeac20b9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s"
Sep 17 21:13:26.036645 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.036584    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wqvk\" (UniqueName: \"kubernetes.io/projected/762d65aa-f29f-4025-a118-c61ceeac20b9-kube-api-access-8wqvk\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s\" (UID: \"762d65aa-f29f-4025-a118-c61ceeac20b9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s"
Sep 17 21:13:26.036645 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.036611    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/762d65aa-f29f-4025-a118-c61ceeac20b9-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s\" (UID: \"762d65aa-f29f-4025-a118-c61ceeac20b9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s"
Sep 17 21:13:26.045514 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.045491    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd"]
Sep 17 21:13:26.048808 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.048790    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd"
Sep 17 21:13:26.051632 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.051616    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 21:13:26.060488 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.060469    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd"]
Sep 17 21:13:26.137426 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.137399    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f57872e-bcb1-4648-895a-320034d57ac4-cabundle-cert\") pod \"isvc-canary-promote-predictor-78b4c78fb9-m4vvd\" (UID: \"7f57872e-bcb1-4648-895a-320034d57ac4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd"
Sep 17 21:13:26.137567 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.137432    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ntvs\" (UniqueName: \"kubernetes.io/projected/7f57872e-bcb1-4648-895a-320034d57ac4-kube-api-access-4ntvs\") pod \"isvc-canary-promote-predictor-78b4c78fb9-m4vvd\" (UID: \"7f57872e-bcb1-4648-895a-320034d57ac4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd"
Sep 17 21:13:26.137567 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.137477    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wqvk\" (UniqueName: \"kubernetes.io/projected/762d65aa-f29f-4025-a118-c61ceeac20b9-kube-api-access-8wqvk\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s\" (UID: \"762d65aa-f29f-4025-a118-c61ceeac20b9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s"
Sep 17 21:13:26.137567 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.137503    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/762d65aa-f29f-4025-a118-c61ceeac20b9-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s\" (UID: \"762d65aa-f29f-4025-a118-c61ceeac20b9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s"
Sep 17 21:13:26.137567 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.137521    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f57872e-bcb1-4648-895a-320034d57ac4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-78b4c78fb9-m4vvd\" (UID: \"7f57872e-bcb1-4648-895a-320034d57ac4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd"
Sep 17 21:13:26.137567 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.137547    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f57872e-bcb1-4648-895a-320034d57ac4-kserve-provision-location\") pod \"isvc-canary-promote-predictor-78b4c78fb9-m4vvd\" (UID: \"7f57872e-bcb1-4648-895a-320034d57ac4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd"
Sep 17 21:13:26.137783 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.137626    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/762d65aa-f29f-4025-a118-c61ceeac20b9-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s\" (UID: \"762d65aa-f29f-4025-a118-c61ceeac20b9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s"
Sep 17 21:13:26.137783 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.137662    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f57872e-bcb1-4648-895a-320034d57ac4-proxy-tls\") pod \"isvc-canary-promote-predictor-78b4c78fb9-m4vvd\" (UID: \"7f57872e-bcb1-4648-895a-320034d57ac4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd"
Sep 17 21:13:26.137783 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.137696    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/762d65aa-f29f-4025-a118-c61ceeac20b9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s\" (UID: \"762d65aa-f29f-4025-a118-c61ceeac20b9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s"
Sep 17 21:13:26.137783 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.137725    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/762d65aa-f29f-4025-a118-c61ceeac20b9-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s\" (UID: \"762d65aa-f29f-4025-a118-c61ceeac20b9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s"
Sep 17 21:13:26.138060 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.138042    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/762d65aa-f29f-4025-a118-c61ceeac20b9-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s\" (UID: \"762d65aa-f29f-4025-a118-c61ceeac20b9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s"
Sep 17 21:13:26.138260 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.138234    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/762d65aa-f29f-4025-a118-c61ceeac20b9-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s\" (UID: \"762d65aa-f29f-4025-a118-c61ceeac20b9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s"
Sep 17 21:13:26.138365 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.138337    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/762d65aa-f29f-4025-a118-c61ceeac20b9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s\" (UID: \"762d65aa-f29f-4025-a118-c61ceeac20b9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s"
Sep 17 21:13:26.140279 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.140259    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/762d65aa-f29f-4025-a118-c61ceeac20b9-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s\" (UID: \"762d65aa-f29f-4025-a118-c61ceeac20b9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s"
Sep 17 21:13:26.148486 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.148468    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wqvk\" (UniqueName: \"kubernetes.io/projected/762d65aa-f29f-4025-a118-c61ceeac20b9-kube-api-access-8wqvk\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s\" (UID: \"762d65aa-f29f-4025-a118-c61ceeac20b9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s"
Sep 17 21:13:26.238812 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.238792    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f57872e-bcb1-4648-895a-320034d57ac4-kserve-provision-location\") pod \"isvc-canary-promote-predictor-78b4c78fb9-m4vvd\" (UID: \"7f57872e-bcb1-4648-895a-320034d57ac4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd"
Sep 17 21:13:26.238935 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.238825    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f57872e-bcb1-4648-895a-320034d57ac4-proxy-tls\") pod \"isvc-canary-promote-predictor-78b4c78fb9-m4vvd\" (UID: \"7f57872e-bcb1-4648-895a-320034d57ac4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd"
Sep 17 21:13:26.238935 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.238850    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f57872e-bcb1-4648-895a-320034d57ac4-cabundle-cert\") pod \"isvc-canary-promote-predictor-78b4c78fb9-m4vvd\" (UID: \"7f57872e-bcb1-4648-895a-320034d57ac4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd"
Sep 17 21:13:26.238935 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.238868    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ntvs\" (UniqueName: \"kubernetes.io/projected/7f57872e-bcb1-4648-895a-320034d57ac4-kube-api-access-4ntvs\") pod \"isvc-canary-promote-predictor-78b4c78fb9-m4vvd\" (UID: \"7f57872e-bcb1-4648-895a-320034d57ac4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd"
Sep 17 21:13:26.238935 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.238903    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f57872e-bcb1-4648-895a-320034d57ac4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-78b4c78fb9-m4vvd\" (UID: \"7f57872e-bcb1-4648-895a-320034d57ac4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd"
Sep 17 21:13:26.239301 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.239192    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f57872e-bcb1-4648-895a-320034d57ac4-kserve-provision-location\") pod \"isvc-canary-promote-predictor-78b4c78fb9-m4vvd\" (UID: \"7f57872e-bcb1-4648-895a-320034d57ac4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd"
Sep 17 21:13:26.239509 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.239493    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f57872e-bcb1-4648-895a-320034d57ac4-cabundle-cert\") pod \"isvc-canary-promote-predictor-78b4c78fb9-m4vvd\" (UID: \"7f57872e-bcb1-4648-895a-320034d57ac4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd"
Sep 17 21:13:26.239557 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.239509    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f57872e-bcb1-4648-895a-320034d57ac4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-78b4c78fb9-m4vvd\" (UID: \"7f57872e-bcb1-4648-895a-320034d57ac4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd"
Sep 17 21:13:26.241098 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.241076    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f57872e-bcb1-4648-895a-320034d57ac4-proxy-tls\") pod \"isvc-canary-promote-predictor-78b4c78fb9-m4vvd\" (UID: \"7f57872e-bcb1-4648-895a-320034d57ac4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd"
Sep 17 21:13:26.247841 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.247816    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ntvs\" (UniqueName: \"kubernetes.io/projected/7f57872e-bcb1-4648-895a-320034d57ac4-kube-api-access-4ntvs\") pod \"isvc-canary-promote-predictor-78b4c78fb9-m4vvd\" (UID: \"7f57872e-bcb1-4648-895a-320034d57ac4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd"
Sep 17 21:13:26.258881 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.258866    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s"
Sep 17 21:13:26.324646 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.324610    2566 generic.go:356] "Generic (PLEG): container finished" podID="1b7d2fc8-6813-41ec-9b52-bae5676260de" containerID="314b2c471874969e46158e57b32f3f7292ce75f9013041dbd8eed8f6ec58f4d2" exitCode=0
Sep 17 21:13:26.324770 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.324662    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7"
Sep 17 21:13:26.324770 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.324687    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7" event={"ID":"1b7d2fc8-6813-41ec-9b52-bae5676260de","Type":"ContainerDied","Data":"314b2c471874969e46158e57b32f3f7292ce75f9013041dbd8eed8f6ec58f4d2"}
Sep 17 21:13:26.324770 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.324745    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7" event={"ID":"1b7d2fc8-6813-41ec-9b52-bae5676260de","Type":"ContainerDied","Data":"eed1f7805c0a7d888d52309ee9040ef721da85bf3cc2371937f8be41578b6d96"}
Sep 17 21:13:26.324877 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.324774    2566 scope.go:117] "RemoveContainer" containerID="8eee3c2a7e7f242701c7191c3b38807eff52c63dd7a7b0e287dd972053c0513d"
Sep 17 21:13:26.334557 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.334540    2566 scope.go:117] "RemoveContainer" containerID="314b2c471874969e46158e57b32f3f7292ce75f9013041dbd8eed8f6ec58f4d2"
Sep 17 21:13:26.341968 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.341951    2566 scope.go:117] "RemoveContainer" containerID="eeecb0b3d8e1b46d4371d800f867c17fe500ec97b454865d51e0430ba470486a"
Sep 17 21:13:26.347285 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.347266    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7"]
Sep 17 21:13:26.349178 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.349146    2566 scope.go:117] "RemoveContainer" containerID="8eee3c2a7e7f242701c7191c3b38807eff52c63dd7a7b0e287dd972053c0513d"
Sep 17 21:13:26.349654 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:13:26.349612    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eee3c2a7e7f242701c7191c3b38807eff52c63dd7a7b0e287dd972053c0513d\": container with ID starting with 8eee3c2a7e7f242701c7191c3b38807eff52c63dd7a7b0e287dd972053c0513d not found: ID does not exist" containerID="8eee3c2a7e7f242701c7191c3b38807eff52c63dd7a7b0e287dd972053c0513d"
Sep 17 21:13:26.349761 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.349668    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eee3c2a7e7f242701c7191c3b38807eff52c63dd7a7b0e287dd972053c0513d"} err="failed to get container status \"8eee3c2a7e7f242701c7191c3b38807eff52c63dd7a7b0e287dd972053c0513d\": rpc error: code = NotFound desc = could not find container \"8eee3c2a7e7f242701c7191c3b38807eff52c63dd7a7b0e287dd972053c0513d\": container with ID starting with 8eee3c2a7e7f242701c7191c3b38807eff52c63dd7a7b0e287dd972053c0513d not found: ID does not exist"
Sep 17 21:13:26.349761 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.349694    2566 scope.go:117] "RemoveContainer" containerID="314b2c471874969e46158e57b32f3f7292ce75f9013041dbd8eed8f6ec58f4d2"
Sep 17 21:13:26.350032 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:13:26.350001    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"314b2c471874969e46158e57b32f3f7292ce75f9013041dbd8eed8f6ec58f4d2\": container with ID starting with 314b2c471874969e46158e57b32f3f7292ce75f9013041dbd8eed8f6ec58f4d2 not found: ID does not exist" containerID="314b2c471874969e46158e57b32f3f7292ce75f9013041dbd8eed8f6ec58f4d2"
Sep 17 21:13:26.350032 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.350029    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314b2c471874969e46158e57b32f3f7292ce75f9013041dbd8eed8f6ec58f4d2"} err="failed to get container status \"314b2c471874969e46158e57b32f3f7292ce75f9013041dbd8eed8f6ec58f4d2\": rpc error: code = NotFound desc = could not find container \"314b2c471874969e46158e57b32f3f7292ce75f9013041dbd8eed8f6ec58f4d2\": container with ID starting with 314b2c471874969e46158e57b32f3f7292ce75f9013041dbd8eed8f6ec58f4d2 not found: ID does not exist"
Sep 17 21:13:26.350199 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.350045    2566 scope.go:117] "RemoveContainer" containerID="eeecb0b3d8e1b46d4371d800f867c17fe500ec97b454865d51e0430ba470486a"
Sep 17 21:13:26.350473 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:13:26.350453    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeecb0b3d8e1b46d4371d800f867c17fe500ec97b454865d51e0430ba470486a\": container with ID starting with eeecb0b3d8e1b46d4371d800f867c17fe500ec97b454865d51e0430ba470486a not found: ID does not exist" containerID="eeecb0b3d8e1b46d4371d800f867c17fe500ec97b454865d51e0430ba470486a"
Sep 17 21:13:26.350539 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.350477    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeecb0b3d8e1b46d4371d800f867c17fe500ec97b454865d51e0430ba470486a"} err="failed to get container status \"eeecb0b3d8e1b46d4371d800f867c17fe500ec97b454865d51e0430ba470486a\": rpc error: code = NotFound desc = could not find container \"eeecb0b3d8e1b46d4371d800f867c17fe500ec97b454865d51e0430ba470486a\": container with ID starting with eeecb0b3d8e1b46d4371d800f867c17fe500ec97b454865d51e0430ba470486a not found: ID does not exist"
Sep 17 21:13:26.351510 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.351493    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-tq6h7"]
Sep 17 21:13:26.358758 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.358740    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd"
Sep 17 21:13:26.458347 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.458318    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b7d2fc8-6813-41ec-9b52-bae5676260de" path="/var/lib/kubelet/pods/1b7d2fc8-6813-41ec-9b52-bae5676260de/volumes"
Sep 17 21:13:26.528422 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.528300    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s"]
Sep 17 21:13:26.530858 ip-10-0-138-140 kubenswrapper[2566]: W0917 21:13:26.530835    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod762d65aa_f29f_4025_a118_c61ceeac20b9.slice/crio-eb3afd4323acf458f2f6610fd9c557d340e2ab2a91362615b3b275f7585c3197 WatchSource:0}: Error finding container eb3afd4323acf458f2f6610fd9c557d340e2ab2a91362615b3b275f7585c3197: Status 404 returned error can't find the container with id eb3afd4323acf458f2f6610fd9c557d340e2ab2a91362615b3b275f7585c3197
Sep 17 21:13:26.628968 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:26.628943    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd"]
Sep 17 21:13:26.631289 ip-10-0-138-140 kubenswrapper[2566]: W0917 21:13:26.631257    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f57872e_bcb1_4648_895a_320034d57ac4.slice/crio-1e6cb659b625f2d1138b07c359d5d01afb02772c75e50ac504c511e169c48c0f WatchSource:0}: Error finding container 1e6cb659b625f2d1138b07c359d5d01afb02772c75e50ac504c511e169c48c0f: Status 404 returned error can't find the container with id 1e6cb659b625f2d1138b07c359d5d01afb02772c75e50ac504c511e169c48c0f
Sep 17 21:13:27.330343 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:27.330310    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd" event={"ID":"7f57872e-bcb1-4648-895a-320034d57ac4","Type":"ContainerStarted","Data":"8ed1257e1cd836060387c72df4148f8fb9a87e48e8c12be8ec75bb7cf4167508"}
Sep 17 21:13:27.330546 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:27.330411    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd" event={"ID":"7f57872e-bcb1-4648-895a-320034d57ac4","Type":"ContainerStarted","Data":"1e6cb659b625f2d1138b07c359d5d01afb02772c75e50ac504c511e169c48c0f"}
Sep 17 21:13:27.331833 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:27.331807    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s" event={"ID":"762d65aa-f29f-4025-a118-c61ceeac20b9","Type":"ContainerStarted","Data":"0c4c9d5803222783f9a9c93a22e97a7ece2b10e91403b3c7c2bfc2eff4f0037d"}
Sep 17 21:13:27.331928 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:27.331861    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s" event={"ID":"762d65aa-f29f-4025-a118-c61ceeac20b9","Type":"ContainerStarted","Data":"eb3afd4323acf458f2f6610fd9c557d340e2ab2a91362615b3b275f7585c3197"}
Sep 17 21:13:31.348499 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:31.348465    2566 generic.go:356] "Generic (PLEG): container finished" podID="7f57872e-bcb1-4648-895a-320034d57ac4" containerID="8ed1257e1cd836060387c72df4148f8fb9a87e48e8c12be8ec75bb7cf4167508" exitCode=0
Sep 17 21:13:31.349031 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:31.348545    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd" event={"ID":"7f57872e-bcb1-4648-895a-320034d57ac4","Type":"ContainerDied","Data":"8ed1257e1cd836060387c72df4148f8fb9a87e48e8c12be8ec75bb7cf4167508"}
Sep 17 21:13:31.350152 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:31.350131    2566 generic.go:356] "Generic (PLEG): container finished" podID="762d65aa-f29f-4025-a118-c61ceeac20b9" containerID="0c4c9d5803222783f9a9c93a22e97a7ece2b10e91403b3c7c2bfc2eff4f0037d" exitCode=0
Sep 17 21:13:31.350250 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:31.350199    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s" event={"ID":"762d65aa-f29f-4025-a118-c61ceeac20b9","Type":"ContainerDied","Data":"0c4c9d5803222783f9a9c93a22e97a7ece2b10e91403b3c7c2bfc2eff4f0037d"}
Sep 17 21:13:32.356375 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:32.356337    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s" event={"ID":"762d65aa-f29f-4025-a118-c61ceeac20b9","Type":"ContainerStarted","Data":"62b2084bbfdf14a85c7aa7868573a0b770c31aa442977de51cede9b1f8732000"}
Sep 17 21:13:32.356800 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:32.356419    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s" event={"ID":"762d65aa-f29f-4025-a118-c61ceeac20b9","Type":"ContainerStarted","Data":"c125dc9067463ca4ccb02792951d73b0f9c8ce1ce870eadaa95d88f7a309e891"}
Sep 17 21:13:32.356800 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:32.356638    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s"
Sep 17 21:13:32.358438 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:32.358414    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd" event={"ID":"7f57872e-bcb1-4648-895a-320034d57ac4","Type":"ContainerStarted","Data":"562a756e21a3961db8c12a6daa402a285a4d0f24fd08e500aa58efcd9b8e1d4e"}
Sep 17 21:13:32.358558 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:32.358459    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd" event={"ID":"7f57872e-bcb1-4648-895a-320034d57ac4","Type":"ContainerStarted","Data":"ed3b8714a3be1fcef4123a0b9cb0148c1c610873f115ab404d50d093d00b5785"}
Sep 17 21:13:32.358705 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:32.358688    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd"
Sep 17 21:13:32.358772 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:32.358729    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd"
Sep 17 21:13:32.360038 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:32.360015    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd" podUID="7f57872e-bcb1-4648-895a-320034d57ac4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 21:13:32.377481 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:32.377438    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s" podStartSLOduration=3.415224856 podStartE2EDuration="7.377424878s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.962200022s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:13:32.375802547 +0000 UTC m=+1453.986535800" watchObservedRunningTime="2026-09-17 21:13:32.377424878 +0000 UTC m=+1453.988158106"
Sep 17 21:13:32.394041 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:32.394002    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd" podStartSLOduration=2.593869387 podStartE2EDuration="6.393991107s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.80012172s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:13:32.393261016 +0000 UTC m=+1454.003994247" watchObservedRunningTime="2026-09-17 21:13:32.393991107 +0000 UTC m=+1454.004724336"
Sep 17 21:13:33.362849 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:33.362811    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s"
Sep 17 21:13:33.363303 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:33.362846    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd" podUID="7f57872e-bcb1-4648-895a-320034d57ac4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 21:13:33.363949 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:33.363926    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s" podUID="762d65aa-f29f-4025-a118-c61ceeac20b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 21:13:34.366575 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:34.366534    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s" podUID="762d65aa-f29f-4025-a118-c61ceeac20b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 21:13:38.368051 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:38.368022    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd"
Sep 17 21:13:38.368515 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:38.368488    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd" podUID="7f57872e-bcb1-4648-895a-320034d57ac4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 21:13:39.371103 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:39.371073    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s"
Sep 17 21:13:39.371540 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:39.371518    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s" podUID="762d65aa-f29f-4025-a118-c61ceeac20b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 21:13:48.369435 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:48.369397    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd" podUID="7f57872e-bcb1-4648-895a-320034d57ac4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 21:13:49.371527 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:49.371492    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s" podUID="762d65aa-f29f-4025-a118-c61ceeac20b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 21:13:58.369441 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:58.369399    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd" podUID="7f57872e-bcb1-4648-895a-320034d57ac4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 21:13:59.371731 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:13:59.371691    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s" podUID="762d65aa-f29f-4025-a118-c61ceeac20b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 21:14:08.369078 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:08.369035    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd" podUID="7f57872e-bcb1-4648-895a-320034d57ac4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 21:14:09.371461 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:09.371420    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s" podUID="762d65aa-f29f-4025-a118-c61ceeac20b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 21:14:18.368528 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:18.368485    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd" podUID="7f57872e-bcb1-4648-895a-320034d57ac4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 21:14:19.371669 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:19.371636    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s" podUID="762d65aa-f29f-4025-a118-c61ceeac20b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 21:14:20.473386 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:20.473360    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t2rlv_ed6472a6-7979-43b0-a549-7ca5c5d8fb0f/console-operator/1.log"
Sep 17 21:14:20.473890 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:20.473872    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t2rlv_ed6472a6-7979-43b0-a549-7ca5c5d8fb0f/console-operator/1.log"
Sep 17 21:14:20.478589 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:20.478566    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnqx7_402f55a4-24b6-47c6-b719-9c6a5b473434/ovn-acl-logging/0.log"
Sep 17 21:14:20.479237 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:20.479205    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnqx7_402f55a4-24b6-47c6-b719-9c6a5b473434/ovn-acl-logging/0.log"
Sep 17 21:14:28.369463 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:28.369421    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd" podUID="7f57872e-bcb1-4648-895a-320034d57ac4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 21:14:29.371467 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:29.371426    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s" podUID="762d65aa-f29f-4025-a118-c61ceeac20b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 21:14:38.368915 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:38.368836    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd" podUID="7f57872e-bcb1-4648-895a-320034d57ac4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 21:14:39.372346 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:39.372309    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s" podUID="762d65aa-f29f-4025-a118-c61ceeac20b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 21:14:42.454724 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:42.454680    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd" podUID="7f57872e-bcb1-4648-895a-320034d57ac4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 21:14:49.372844 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:49.372812    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s"
Sep 17 21:14:49.515485 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:49.515450    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd"]
Sep 17 21:14:49.515775 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:49.515737    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd" podUID="7f57872e-bcb1-4648-895a-320034d57ac4" containerName="kserve-container" containerID="cri-o://ed3b8714a3be1fcef4123a0b9cb0148c1c610873f115ab404d50d093d00b5785" gracePeriod=30
Sep 17 21:14:49.515919 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:49.515737    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd" podUID="7f57872e-bcb1-4648-895a-320034d57ac4" containerName="kube-rbac-proxy" containerID="cri-o://562a756e21a3961db8c12a6daa402a285a4d0f24fd08e500aa58efcd9b8e1d4e" gracePeriod=30
Sep 17 21:14:49.516546 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:49.516386    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd"
Sep 17 21:14:49.666174 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:49.666143    2566 generic.go:356] "Generic (PLEG): container finished" podID="7f57872e-bcb1-4648-895a-320034d57ac4" containerID="562a756e21a3961db8c12a6daa402a285a4d0f24fd08e500aa58efcd9b8e1d4e" exitCode=2
Sep 17 21:14:49.666341 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:49.666201    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd" event={"ID":"7f57872e-bcb1-4648-895a-320034d57ac4","Type":"ContainerDied","Data":"562a756e21a3961db8c12a6daa402a285a4d0f24fd08e500aa58efcd9b8e1d4e"}
Sep 17 21:14:53.363634 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:53.363592    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd" podUID="7f57872e-bcb1-4648-895a-320034d57ac4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 17 21:14:54.154527 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.154504    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd"
Sep 17 21:14:54.230176 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.230144    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ntvs\" (UniqueName: \"kubernetes.io/projected/7f57872e-bcb1-4648-895a-320034d57ac4-kube-api-access-4ntvs\") pod \"7f57872e-bcb1-4648-895a-320034d57ac4\" (UID: \"7f57872e-bcb1-4648-895a-320034d57ac4\") "
Sep 17 21:14:54.230359 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.230187    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f57872e-bcb1-4648-895a-320034d57ac4-cabundle-cert\") pod \"7f57872e-bcb1-4648-895a-320034d57ac4\" (UID: \"7f57872e-bcb1-4648-895a-320034d57ac4\") "
Sep 17 21:14:54.230359 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.230320    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f57872e-bcb1-4648-895a-320034d57ac4-kserve-provision-location\") pod \"7f57872e-bcb1-4648-895a-320034d57ac4\" (UID: \"7f57872e-bcb1-4648-895a-320034d57ac4\") "
Sep 17 21:14:54.230359 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.230357    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f57872e-bcb1-4648-895a-320034d57ac4-kube-rbac-proxy-sar-config\") pod \"7f57872e-bcb1-4648-895a-320034d57ac4\" (UID: \"7f57872e-bcb1-4648-895a-320034d57ac4\") "
Sep 17 21:14:54.230631 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.230405    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f57872e-bcb1-4648-895a-320034d57ac4-proxy-tls\") pod \"7f57872e-bcb1-4648-895a-320034d57ac4\" (UID: \"7f57872e-bcb1-4648-895a-320034d57ac4\") "
Sep 17 21:14:54.230687 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.230626    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f57872e-bcb1-4648-895a-320034d57ac4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7f57872e-bcb1-4648-895a-320034d57ac4" (UID: "7f57872e-bcb1-4648-895a-320034d57ac4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:14:54.230739 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.230696    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f57872e-bcb1-4648-895a-320034d57ac4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7f57872e-bcb1-4648-895a-320034d57ac4" (UID: "7f57872e-bcb1-4648-895a-320034d57ac4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:14:54.230777 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.230764    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f57872e-bcb1-4648-895a-320034d57ac4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7f57872e-bcb1-4648-895a-320034d57ac4" (UID: "7f57872e-bcb1-4648-895a-320034d57ac4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:14:54.232441 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.232420    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f57872e-bcb1-4648-895a-320034d57ac4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7f57872e-bcb1-4648-895a-320034d57ac4" (UID: "7f57872e-bcb1-4648-895a-320034d57ac4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:14:54.232500 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.232436    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f57872e-bcb1-4648-895a-320034d57ac4-kube-api-access-4ntvs" (OuterVolumeSpecName: "kube-api-access-4ntvs") pod "7f57872e-bcb1-4648-895a-320034d57ac4" (UID: "7f57872e-bcb1-4648-895a-320034d57ac4"). InnerVolumeSpecName "kube-api-access-4ntvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:14:54.331377 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.331301    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f57872e-bcb1-4648-895a-320034d57ac4-kserve-provision-location\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:14:54.331377 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.331326    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f57872e-bcb1-4648-895a-320034d57ac4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:14:54.331377 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.331335    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f57872e-bcb1-4648-895a-320034d57ac4-proxy-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:14:54.331377 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.331344    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ntvs\" (UniqueName: \"kubernetes.io/projected/7f57872e-bcb1-4648-895a-320034d57ac4-kube-api-access-4ntvs\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:14:54.331377 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.331352    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f57872e-bcb1-4648-895a-320034d57ac4-cabundle-cert\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:14:54.492153 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:14:54.492125    2566 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f57872e_bcb1_4648_895a_320034d57ac4.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f57872e_bcb1_4648_895a_320034d57ac4.slice/crio-1e6cb659b625f2d1138b07c359d5d01afb02772c75e50ac504c511e169c48c0f\": RecentStats: unable to find data in memory cache]"
Sep 17 21:14:54.686732 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.686652    2566 generic.go:356] "Generic (PLEG): container finished" podID="7f57872e-bcb1-4648-895a-320034d57ac4" containerID="ed3b8714a3be1fcef4123a0b9cb0148c1c610873f115ab404d50d093d00b5785" exitCode=0
Sep 17 21:14:54.686732 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.686705    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd"
Sep 17 21:14:54.686904 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.686732    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd" event={"ID":"7f57872e-bcb1-4648-895a-320034d57ac4","Type":"ContainerDied","Data":"ed3b8714a3be1fcef4123a0b9cb0148c1c610873f115ab404d50d093d00b5785"}
Sep 17 21:14:54.686904 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.686795    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd" event={"ID":"7f57872e-bcb1-4648-895a-320034d57ac4","Type":"ContainerDied","Data":"1e6cb659b625f2d1138b07c359d5d01afb02772c75e50ac504c511e169c48c0f"}
Sep 17 21:14:54.686904 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.686821    2566 scope.go:117] "RemoveContainer" containerID="562a756e21a3961db8c12a6daa402a285a4d0f24fd08e500aa58efcd9b8e1d4e"
Sep 17 21:14:54.694675 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.694659    2566 scope.go:117] "RemoveContainer" containerID="ed3b8714a3be1fcef4123a0b9cb0148c1c610873f115ab404d50d093d00b5785"
Sep 17 21:14:54.701511 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.701495    2566 scope.go:117] "RemoveContainer" containerID="8ed1257e1cd836060387c72df4148f8fb9a87e48e8c12be8ec75bb7cf4167508"
Sep 17 21:14:54.708330 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.708308    2566 scope.go:117] "RemoveContainer" containerID="562a756e21a3961db8c12a6daa402a285a4d0f24fd08e500aa58efcd9b8e1d4e"
Sep 17 21:14:54.708575 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:14:54.708557    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"562a756e21a3961db8c12a6daa402a285a4d0f24fd08e500aa58efcd9b8e1d4e\": container with ID starting with 562a756e21a3961db8c12a6daa402a285a4d0f24fd08e500aa58efcd9b8e1d4e not found: ID does not exist" containerID="562a756e21a3961db8c12a6daa402a285a4d0f24fd08e500aa58efcd9b8e1d4e"
Sep 17 21:14:54.708630 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.708583    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"562a756e21a3961db8c12a6daa402a285a4d0f24fd08e500aa58efcd9b8e1d4e"} err="failed to get container status \"562a756e21a3961db8c12a6daa402a285a4d0f24fd08e500aa58efcd9b8e1d4e\": rpc error: code = NotFound desc = could not find container \"562a756e21a3961db8c12a6daa402a285a4d0f24fd08e500aa58efcd9b8e1d4e\": container with ID starting with 562a756e21a3961db8c12a6daa402a285a4d0f24fd08e500aa58efcd9b8e1d4e not found: ID does not exist"
Sep 17 21:14:54.708630 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.708597    2566 scope.go:117] "RemoveContainer" containerID="ed3b8714a3be1fcef4123a0b9cb0148c1c610873f115ab404d50d093d00b5785"
Sep 17 21:14:54.708856 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:14:54.708836    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed3b8714a3be1fcef4123a0b9cb0148c1c610873f115ab404d50d093d00b5785\": container with ID starting with ed3b8714a3be1fcef4123a0b9cb0148c1c610873f115ab404d50d093d00b5785 not found: ID does not exist" containerID="ed3b8714a3be1fcef4123a0b9cb0148c1c610873f115ab404d50d093d00b5785"
Sep 17 21:14:54.708901 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.708863    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed3b8714a3be1fcef4123a0b9cb0148c1c610873f115ab404d50d093d00b5785"} err="failed to get container status \"ed3b8714a3be1fcef4123a0b9cb0148c1c610873f115ab404d50d093d00b5785\": rpc error: code = NotFound desc = could not find container \"ed3b8714a3be1fcef4123a0b9cb0148c1c610873f115ab404d50d093d00b5785\": container with ID starting with ed3b8714a3be1fcef4123a0b9cb0148c1c610873f115ab404d50d093d00b5785 not found: ID does not exist"
Sep 17 21:14:54.708901 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.708878    2566 scope.go:117] "RemoveContainer" containerID="8ed1257e1cd836060387c72df4148f8fb9a87e48e8c12be8ec75bb7cf4167508"
Sep 17 21:14:54.709122 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:14:54.709108    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ed1257e1cd836060387c72df4148f8fb9a87e48e8c12be8ec75bb7cf4167508\": container with ID starting with 8ed1257e1cd836060387c72df4148f8fb9a87e48e8c12be8ec75bb7cf4167508 not found: ID does not exist" containerID="8ed1257e1cd836060387c72df4148f8fb9a87e48e8c12be8ec75bb7cf4167508"
Sep 17 21:14:54.709174 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.709124    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ed1257e1cd836060387c72df4148f8fb9a87e48e8c12be8ec75bb7cf4167508"} err="failed to get container status \"8ed1257e1cd836060387c72df4148f8fb9a87e48e8c12be8ec75bb7cf4167508\": rpc error: code = NotFound desc = could not find container \"8ed1257e1cd836060387c72df4148f8fb9a87e48e8c12be8ec75bb7cf4167508\": container with ID starting with 8ed1257e1cd836060387c72df4148f8fb9a87e48e8c12be8ec75bb7cf4167508 not found: ID does not exist"
Sep 17 21:14:54.712358 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.712338    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd"]
Sep 17 21:14:54.725004 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:54.724985    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-m4vvd"]
Sep 17 21:14:56.458857 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:14:56.458822    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f57872e-bcb1-4648-895a-320034d57ac4" path="/var/lib/kubelet/pods/7f57872e-bcb1-4648-895a-320034d57ac4/volumes"
Sep 17 21:16:36.387798 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:36.387731    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s"]
Sep 17 21:16:36.388271 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:36.387986    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s" podUID="762d65aa-f29f-4025-a118-c61ceeac20b9" containerName="kserve-container" containerID="cri-o://c125dc9067463ca4ccb02792951d73b0f9c8ce1ce870eadaa95d88f7a309e891" gracePeriod=30
Sep 17 21:16:36.388271 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:36.388029    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s" podUID="762d65aa-f29f-4025-a118-c61ceeac20b9" containerName="kube-rbac-proxy" containerID="cri-o://62b2084bbfdf14a85c7aa7868573a0b770c31aa442977de51cede9b1f8732000" gracePeriod=30
Sep 17 21:16:37.085534 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:37.085501    2566 generic.go:356] "Generic (PLEG): container finished" podID="762d65aa-f29f-4025-a118-c61ceeac20b9" containerID="62b2084bbfdf14a85c7aa7868573a0b770c31aa442977de51cede9b1f8732000" exitCode=2
Sep 17 21:16:37.085694 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:37.085550    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s" event={"ID":"762d65aa-f29f-4025-a118-c61ceeac20b9","Type":"ContainerDied","Data":"62b2084bbfdf14a85c7aa7868573a0b770c31aa442977de51cede9b1f8732000"}
Sep 17 21:16:39.366829 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:39.366791    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s" podUID="762d65aa-f29f-4025-a118-c61ceeac20b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Sep 17 21:16:39.372178 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:39.372139    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s" podUID="762d65aa-f29f-4025-a118-c61ceeac20b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 21:16:40.837736 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:40.837709    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s"
Sep 17 21:16:40.914458 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:40.914382    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/762d65aa-f29f-4025-a118-c61ceeac20b9-proxy-tls\") pod \"762d65aa-f29f-4025-a118-c61ceeac20b9\" (UID: \"762d65aa-f29f-4025-a118-c61ceeac20b9\") "
Sep 17 21:16:40.914458 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:40.914425    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/762d65aa-f29f-4025-a118-c61ceeac20b9-cabundle-cert\") pod \"762d65aa-f29f-4025-a118-c61ceeac20b9\" (UID: \"762d65aa-f29f-4025-a118-c61ceeac20b9\") "
Sep 17 21:16:40.914648 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:40.914479    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/762d65aa-f29f-4025-a118-c61ceeac20b9-kserve-provision-location\") pod \"762d65aa-f29f-4025-a118-c61ceeac20b9\" (UID: \"762d65aa-f29f-4025-a118-c61ceeac20b9\") "
Sep 17 21:16:40.914648 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:40.914499    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wqvk\" (UniqueName: \"kubernetes.io/projected/762d65aa-f29f-4025-a118-c61ceeac20b9-kube-api-access-8wqvk\") pod \"762d65aa-f29f-4025-a118-c61ceeac20b9\" (UID: \"762d65aa-f29f-4025-a118-c61ceeac20b9\") "
Sep 17 21:16:40.914648 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:40.914521    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/762d65aa-f29f-4025-a118-c61ceeac20b9-kube-rbac-proxy-sar-config\") pod \"762d65aa-f29f-4025-a118-c61ceeac20b9\" (UID: \"762d65aa-f29f-4025-a118-c61ceeac20b9\") "
Sep 17 21:16:40.914830 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:40.914805    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762d65aa-f29f-4025-a118-c61ceeac20b9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "762d65aa-f29f-4025-a118-c61ceeac20b9" (UID: "762d65aa-f29f-4025-a118-c61ceeac20b9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:16:40.914888 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:40.914855    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/762d65aa-f29f-4025-a118-c61ceeac20b9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "762d65aa-f29f-4025-a118-c61ceeac20b9" (UID: "762d65aa-f29f-4025-a118-c61ceeac20b9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:16:40.914939 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:40.914899    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/762d65aa-f29f-4025-a118-c61ceeac20b9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "762d65aa-f29f-4025-a118-c61ceeac20b9" (UID: "762d65aa-f29f-4025-a118-c61ceeac20b9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:16:40.916407 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:40.916388    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/762d65aa-f29f-4025-a118-c61ceeac20b9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "762d65aa-f29f-4025-a118-c61ceeac20b9" (UID: "762d65aa-f29f-4025-a118-c61ceeac20b9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:16:40.916543 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:40.916522    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762d65aa-f29f-4025-a118-c61ceeac20b9-kube-api-access-8wqvk" (OuterVolumeSpecName: "kube-api-access-8wqvk") pod "762d65aa-f29f-4025-a118-c61ceeac20b9" (UID: "762d65aa-f29f-4025-a118-c61ceeac20b9"). InnerVolumeSpecName "kube-api-access-8wqvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:16:41.015181 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:41.015156    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/762d65aa-f29f-4025-a118-c61ceeac20b9-proxy-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:16:41.015181 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:41.015177    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/762d65aa-f29f-4025-a118-c61ceeac20b9-cabundle-cert\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:16:41.015359 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:41.015186    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/762d65aa-f29f-4025-a118-c61ceeac20b9-kserve-provision-location\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:16:41.015359 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:41.015196    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wqvk\" (UniqueName: \"kubernetes.io/projected/762d65aa-f29f-4025-a118-c61ceeac20b9-kube-api-access-8wqvk\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:16:41.015359 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:41.015205    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/762d65aa-f29f-4025-a118-c61ceeac20b9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:16:41.106034 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:41.106007    2566 generic.go:356] "Generic (PLEG): container finished" podID="762d65aa-f29f-4025-a118-c61ceeac20b9" containerID="c125dc9067463ca4ccb02792951d73b0f9c8ce1ce870eadaa95d88f7a309e891" exitCode=0
Sep 17 21:16:41.106145 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:41.106065    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s"
Sep 17 21:16:41.106145 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:41.106076    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s" event={"ID":"762d65aa-f29f-4025-a118-c61ceeac20b9","Type":"ContainerDied","Data":"c125dc9067463ca4ccb02792951d73b0f9c8ce1ce870eadaa95d88f7a309e891"}
Sep 17 21:16:41.106145 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:41.106121    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s" event={"ID":"762d65aa-f29f-4025-a118-c61ceeac20b9","Type":"ContainerDied","Data":"eb3afd4323acf458f2f6610fd9c557d340e2ab2a91362615b3b275f7585c3197"}
Sep 17 21:16:41.106269 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:41.106146    2566 scope.go:117] "RemoveContainer" containerID="62b2084bbfdf14a85c7aa7868573a0b770c31aa442977de51cede9b1f8732000"
Sep 17 21:16:41.114716 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:41.114701    2566 scope.go:117] "RemoveContainer" containerID="c125dc9067463ca4ccb02792951d73b0f9c8ce1ce870eadaa95d88f7a309e891"
Sep 17 21:16:41.121748 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:41.121729    2566 scope.go:117] "RemoveContainer" containerID="0c4c9d5803222783f9a9c93a22e97a7ece2b10e91403b3c7c2bfc2eff4f0037d"
Sep 17 21:16:41.127928 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:41.127905    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s"]
Sep 17 21:16:41.129108 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:41.129092    2566 scope.go:117] "RemoveContainer" containerID="62b2084bbfdf14a85c7aa7868573a0b770c31aa442977de51cede9b1f8732000"
Sep 17 21:16:41.129381 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:16:41.129362    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62b2084bbfdf14a85c7aa7868573a0b770c31aa442977de51cede9b1f8732000\": container with ID starting with 62b2084bbfdf14a85c7aa7868573a0b770c31aa442977de51cede9b1f8732000 not found: ID does not exist" containerID="62b2084bbfdf14a85c7aa7868573a0b770c31aa442977de51cede9b1f8732000"
Sep 17 21:16:41.129457 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:41.129392    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b2084bbfdf14a85c7aa7868573a0b770c31aa442977de51cede9b1f8732000"} err="failed to get container status \"62b2084bbfdf14a85c7aa7868573a0b770c31aa442977de51cede9b1f8732000\": rpc error: code = NotFound desc = could not find container \"62b2084bbfdf14a85c7aa7868573a0b770c31aa442977de51cede9b1f8732000\": container with ID starting with 62b2084bbfdf14a85c7aa7868573a0b770c31aa442977de51cede9b1f8732000 not found: ID does not exist"
Sep 17 21:16:41.129457 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:41.129415    2566 scope.go:117] "RemoveContainer" containerID="c125dc9067463ca4ccb02792951d73b0f9c8ce1ce870eadaa95d88f7a309e891"
Sep 17 21:16:41.129676 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:16:41.129658    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c125dc9067463ca4ccb02792951d73b0f9c8ce1ce870eadaa95d88f7a309e891\": container with ID starting with c125dc9067463ca4ccb02792951d73b0f9c8ce1ce870eadaa95d88f7a309e891 not found: ID does not exist" containerID="c125dc9067463ca4ccb02792951d73b0f9c8ce1ce870eadaa95d88f7a309e891"
Sep 17 21:16:41.129714 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:41.129681    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c125dc9067463ca4ccb02792951d73b0f9c8ce1ce870eadaa95d88f7a309e891"} err="failed to get container status \"c125dc9067463ca4ccb02792951d73b0f9c8ce1ce870eadaa95d88f7a309e891\": rpc error: code = NotFound desc = could not find container \"c125dc9067463ca4ccb02792951d73b0f9c8ce1ce870eadaa95d88f7a309e891\": container with ID starting with c125dc9067463ca4ccb02792951d73b0f9c8ce1ce870eadaa95d88f7a309e891 not found: ID does not exist"
Sep 17 21:16:41.129714 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:41.129697    2566 scope.go:117] "RemoveContainer" containerID="0c4c9d5803222783f9a9c93a22e97a7ece2b10e91403b3c7c2bfc2eff4f0037d"
Sep 17 21:16:41.129920 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:16:41.129904    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c4c9d5803222783f9a9c93a22e97a7ece2b10e91403b3c7c2bfc2eff4f0037d\": container with ID starting with 0c4c9d5803222783f9a9c93a22e97a7ece2b10e91403b3c7c2bfc2eff4f0037d not found: ID does not exist" containerID="0c4c9d5803222783f9a9c93a22e97a7ece2b10e91403b3c7c2bfc2eff4f0037d"
Sep 17 21:16:41.129972 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:41.129925    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c4c9d5803222783f9a9c93a22e97a7ece2b10e91403b3c7c2bfc2eff4f0037d"} err="failed to get container status \"0c4c9d5803222783f9a9c93a22e97a7ece2b10e91403b3c7c2bfc2eff4f0037d\": rpc error: code = NotFound desc = could not find container \"0c4c9d5803222783f9a9c93a22e97a7ece2b10e91403b3c7c2bfc2eff4f0037d\": container with ID starting with 0c4c9d5803222783f9a9c93a22e97a7ece2b10e91403b3c7c2bfc2eff4f0037d not found: ID does not exist"
Sep 17 21:16:41.132104 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:41.132085    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-6nx2s"]
Sep 17 21:16:42.458578 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:16:42.458541    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="762d65aa-f29f-4025-a118-c61ceeac20b9" path="/var/lib/kubelet/pods/762d65aa-f29f-4025-a118-c61ceeac20b9/volumes"
Sep 17 21:17:01.132520 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.132484    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"]
Sep 17 21:17:01.132902 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.132781    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="762d65aa-f29f-4025-a118-c61ceeac20b9" containerName="storage-initializer"
Sep 17 21:17:01.132902 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.132793    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="762d65aa-f29f-4025-a118-c61ceeac20b9" containerName="storage-initializer"
Sep 17 21:17:01.132902 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.132801    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f57872e-bcb1-4648-895a-320034d57ac4" containerName="kserve-container"
Sep 17 21:17:01.132902 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.132807    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f57872e-bcb1-4648-895a-320034d57ac4" containerName="kserve-container"
Sep 17 21:17:01.132902 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.132814    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="762d65aa-f29f-4025-a118-c61ceeac20b9" containerName="kube-rbac-proxy"
Sep 17 21:17:01.132902 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.132820    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="762d65aa-f29f-4025-a118-c61ceeac20b9" containerName="kube-rbac-proxy"
Sep 17 21:17:01.132902 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.132834    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f57872e-bcb1-4648-895a-320034d57ac4" containerName="storage-initializer"
Sep 17 21:17:01.132902 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.132842    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f57872e-bcb1-4648-895a-320034d57ac4" containerName="storage-initializer"
Sep 17 21:17:01.132902 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.132853    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="762d65aa-f29f-4025-a118-c61ceeac20b9" containerName="kserve-container"
Sep 17 21:17:01.132902 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.132858    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="762d65aa-f29f-4025-a118-c61ceeac20b9" containerName="kserve-container"
Sep 17 21:17:01.132902 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.132865    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f57872e-bcb1-4648-895a-320034d57ac4" containerName="kube-rbac-proxy"
Sep 17 21:17:01.132902 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.132870    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f57872e-bcb1-4648-895a-320034d57ac4" containerName="kube-rbac-proxy"
Sep 17 21:17:01.133298 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.132919    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f57872e-bcb1-4648-895a-320034d57ac4" containerName="kube-rbac-proxy"
Sep 17 21:17:01.133298 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.132928    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="762d65aa-f29f-4025-a118-c61ceeac20b9" containerName="kserve-container"
Sep 17 21:17:01.133298 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.132935    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f57872e-bcb1-4648-895a-320034d57ac4" containerName="kserve-container"
Sep 17 21:17:01.133298 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.132941    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="762d65aa-f29f-4025-a118-c61ceeac20b9" containerName="kube-rbac-proxy"
Sep 17 21:17:01.136093 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.136078    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"
Sep 17 21:17:01.138948 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.138924    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 21:17:01.139089 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.138959    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 17 21:17:01.139089 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.139010    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5dj6n\""
Sep 17 21:17:01.139089 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.139047    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 21:17:01.139267 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.139247    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:17:01.139906 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.139889    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\""
Sep 17 21:17:01.139950 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.139911    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 21:17:01.146702 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.146683    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"]
Sep 17 21:17:01.162189 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.162168    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a68d8b52-888b-4113-972c-3fc6aa9c7dc8-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-797ff577-vsv2w\" (UID: \"a68d8b52-888b-4113-972c-3fc6aa9c7dc8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"
Sep 17 21:17:01.162304 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.162194    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a68d8b52-888b-4113-972c-3fc6aa9c7dc8-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-797ff577-vsv2w\" (UID: \"a68d8b52-888b-4113-972c-3fc6aa9c7dc8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"
Sep 17 21:17:01.162304 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.162237    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a68d8b52-888b-4113-972c-3fc6aa9c7dc8-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-797ff577-vsv2w\" (UID: \"a68d8b52-888b-4113-972c-3fc6aa9c7dc8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"
Sep 17 21:17:01.162304 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.162291    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7t46\" (UniqueName: \"kubernetes.io/projected/a68d8b52-888b-4113-972c-3fc6aa9c7dc8-kube-api-access-x7t46\") pod \"isvc-logger-raw-cipn-predictor-797ff577-vsv2w\" (UID: \"a68d8b52-888b-4113-972c-3fc6aa9c7dc8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"
Sep 17 21:17:01.162426 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.162327    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a68d8b52-888b-4113-972c-3fc6aa9c7dc8-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-797ff577-vsv2w\" (UID: \"a68d8b52-888b-4113-972c-3fc6aa9c7dc8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"
Sep 17 21:17:01.263492 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.263468    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a68d8b52-888b-4113-972c-3fc6aa9c7dc8-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-797ff577-vsv2w\" (UID: \"a68d8b52-888b-4113-972c-3fc6aa9c7dc8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"
Sep 17 21:17:01.263492 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.263497    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a68d8b52-888b-4113-972c-3fc6aa9c7dc8-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-797ff577-vsv2w\" (UID: \"a68d8b52-888b-4113-972c-3fc6aa9c7dc8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"
Sep 17 21:17:01.263699 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.263522    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a68d8b52-888b-4113-972c-3fc6aa9c7dc8-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-797ff577-vsv2w\" (UID: \"a68d8b52-888b-4113-972c-3fc6aa9c7dc8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"
Sep 17 21:17:01.263699 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.263550    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7t46\" (UniqueName: \"kubernetes.io/projected/a68d8b52-888b-4113-972c-3fc6aa9c7dc8-kube-api-access-x7t46\") pod \"isvc-logger-raw-cipn-predictor-797ff577-vsv2w\" (UID: \"a68d8b52-888b-4113-972c-3fc6aa9c7dc8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"
Sep 17 21:17:01.263699 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.263587    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a68d8b52-888b-4113-972c-3fc6aa9c7dc8-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-797ff577-vsv2w\" (UID: \"a68d8b52-888b-4113-972c-3fc6aa9c7dc8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"
Sep 17 21:17:01.263986 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.263966    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a68d8b52-888b-4113-972c-3fc6aa9c7dc8-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-797ff577-vsv2w\" (UID: \"a68d8b52-888b-4113-972c-3fc6aa9c7dc8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"
Sep 17 21:17:01.264413 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.264393    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a68d8b52-888b-4113-972c-3fc6aa9c7dc8-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-797ff577-vsv2w\" (UID: \"a68d8b52-888b-4113-972c-3fc6aa9c7dc8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"
Sep 17 21:17:01.264474 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.264393    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a68d8b52-888b-4113-972c-3fc6aa9c7dc8-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-797ff577-vsv2w\" (UID: \"a68d8b52-888b-4113-972c-3fc6aa9c7dc8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"
Sep 17 21:17:01.265918 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.265900    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a68d8b52-888b-4113-972c-3fc6aa9c7dc8-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-797ff577-vsv2w\" (UID: \"a68d8b52-888b-4113-972c-3fc6aa9c7dc8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"
Sep 17 21:17:01.271906 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.271884    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7t46\" (UniqueName: \"kubernetes.io/projected/a68d8b52-888b-4113-972c-3fc6aa9c7dc8-kube-api-access-x7t46\") pod \"isvc-logger-raw-cipn-predictor-797ff577-vsv2w\" (UID: \"a68d8b52-888b-4113-972c-3fc6aa9c7dc8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"
Sep 17 21:17:01.446968 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.446891    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"
Sep 17 21:17:01.716050 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:01.715916    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"]
Sep 17 21:17:01.718628 ip-10-0-138-140 kubenswrapper[2566]: W0917 21:17:01.718600    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda68d8b52_888b_4113_972c_3fc6aa9c7dc8.slice/crio-de25776f20847747c2adbf98a45324b9d1fd2907cc0fcd7547eba3e227111271 WatchSource:0}: Error finding container de25776f20847747c2adbf98a45324b9d1fd2907cc0fcd7547eba3e227111271: Status 404 returned error can't find the container with id de25776f20847747c2adbf98a45324b9d1fd2907cc0fcd7547eba3e227111271
Sep 17 21:17:02.189441 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:02.189361    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" event={"ID":"a68d8b52-888b-4113-972c-3fc6aa9c7dc8","Type":"ContainerStarted","Data":"3386af51d158ffe757dc38edb6e424ce92a667eab57d24d070be4c48c75a2333"}
Sep 17 21:17:02.189441 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:02.189419    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" event={"ID":"a68d8b52-888b-4113-972c-3fc6aa9c7dc8","Type":"ContainerStarted","Data":"de25776f20847747c2adbf98a45324b9d1fd2907cc0fcd7547eba3e227111271"}
Sep 17 21:17:06.205488 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:06.205454    2566 generic.go:356] "Generic (PLEG): container finished" podID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerID="3386af51d158ffe757dc38edb6e424ce92a667eab57d24d070be4c48c75a2333" exitCode=0
Sep 17 21:17:06.205908 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:06.205498    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" event={"ID":"a68d8b52-888b-4113-972c-3fc6aa9c7dc8","Type":"ContainerDied","Data":"3386af51d158ffe757dc38edb6e424ce92a667eab57d24d070be4c48c75a2333"}
Sep 17 21:17:07.211805 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:07.211769    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" event={"ID":"a68d8b52-888b-4113-972c-3fc6aa9c7dc8","Type":"ContainerStarted","Data":"326e1261ecf60dd08ae23a45e9e79db762bdbda8495acf82e6d9aecf6c8277a4"}
Sep 17 21:17:07.212191 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:07.211840    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" event={"ID":"a68d8b52-888b-4113-972c-3fc6aa9c7dc8","Type":"ContainerStarted","Data":"eb46fde6db72c635b10808624c17800ee4518ab1e1ffacb5ef68dbf73cf4f5d0"}
Sep 17 21:17:07.212191 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:07.211866    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" event={"ID":"a68d8b52-888b-4113-972c-3fc6aa9c7dc8","Type":"ContainerStarted","Data":"89305c22b136737018182cf252d0a32f539048e0a062cc5f0a40f62a15920fa0"}
Sep 17 21:17:07.212191 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:07.212156    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"
Sep 17 21:17:07.212343 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:07.212254    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"
Sep 17 21:17:07.213573 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:07.213549    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 21:17:07.235488 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:07.235438    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podStartSLOduration=2.587067444 podStartE2EDuration="6.235419826s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.648352382s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:17:07.234060446 +0000 UTC m=+1668.844793676" watchObservedRunningTime="2026-09-17 21:17:07.235419826 +0000 UTC m=+1668.846153062"
Sep 17 21:17:08.216240 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:08.216191    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"
Sep 17 21:17:08.216638 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:08.216383    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 21:17:08.217498 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:08.217472    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:17:09.219667 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:09.219621    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 21:17:09.220114 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:09.220086    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:17:14.224052 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:14.224026    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"
Sep 17 21:17:14.224713 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:14.224687    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 21:17:14.224983 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:14.224955    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:17:24.224936 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:24.224894    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 21:17:24.225466 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:24.225440    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:17:34.225081 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:34.224993    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 21:17:34.225526 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:34.225497    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:17:44.225026 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:44.224978    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 21:17:44.225510 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:44.225452    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:17:54.225408 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:54.225364    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 21:17:54.225954 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:17:54.225832    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:18:04.224603 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:04.224567    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 21:18:04.225092 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:04.225066    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:18:14.225129 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:14.225081    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 21:18:14.225612 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:14.225563    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:18:21.454875 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:21.454847    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"
Sep 17 21:18:21.455304 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:21.455168    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"
Sep 17 21:18:36.915354 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:36.915318    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"]
Sep 17 21:18:36.915867 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:36.915569    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="kserve-container" containerID="cri-o://89305c22b136737018182cf252d0a32f539048e0a062cc5f0a40f62a15920fa0" gracePeriod=30
Sep 17 21:18:36.915867 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:36.915622    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="kube-rbac-proxy" containerID="cri-o://eb46fde6db72c635b10808624c17800ee4518ab1e1ffacb5ef68dbf73cf4f5d0" gracePeriod=30
Sep 17 21:18:36.915867 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:36.915622    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="agent" containerID="cri-o://326e1261ecf60dd08ae23a45e9e79db762bdbda8495acf82e6d9aecf6c8277a4" gracePeriod=30
Sep 17 21:18:37.560497 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:37.560464    2566 generic.go:356] "Generic (PLEG): container finished" podID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerID="eb46fde6db72c635b10808624c17800ee4518ab1e1ffacb5ef68dbf73cf4f5d0" exitCode=2
Sep 17 21:18:37.560665 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:37.560536    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" event={"ID":"a68d8b52-888b-4113-972c-3fc6aa9c7dc8","Type":"ContainerDied","Data":"eb46fde6db72c635b10808624c17800ee4518ab1e1ffacb5ef68dbf73cf4f5d0"}
Sep 17 21:18:39.220709 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:39.220658    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 17 21:18:41.454742 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:41.454695    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 21:18:41.455123 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:41.455081    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:18:41.579336 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:41.579266    2566 generic.go:356] "Generic (PLEG): container finished" podID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerID="89305c22b136737018182cf252d0a32f539048e0a062cc5f0a40f62a15920fa0" exitCode=0
Sep 17 21:18:41.579336 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:41.579318    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" event={"ID":"a68d8b52-888b-4113-972c-3fc6aa9c7dc8","Type":"ContainerDied","Data":"89305c22b136737018182cf252d0a32f539048e0a062cc5f0a40f62a15920fa0"}
Sep 17 21:18:44.220158 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:44.220109    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 17 21:18:49.219851 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:49.219808    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 17 21:18:49.220328 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:49.219941    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"
Sep 17 21:18:51.455345 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:51.455303    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 21:18:51.455766 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:51.455643    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:18:53.652220 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:53.652180    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/kserve-container/0.log"
Sep 17 21:18:53.675655 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:53.675632    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/kube-rbac-proxy/0.log"
Sep 17 21:18:53.688294 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:53.688271    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/agent/0.log"
Sep 17 21:18:53.700201 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:53.700182    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/storage-initializer/0.log"
Sep 17 21:18:54.219837 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:54.219799    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 17 21:18:54.561354 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:54.561282    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/kserve-container/0.log"
Sep 17 21:18:54.572676 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:54.572651    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/kube-rbac-proxy/0.log"
Sep 17 21:18:54.582879 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:54.582855    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/agent/0.log"
Sep 17 21:18:54.592776 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:54.592754    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/storage-initializer/0.log"
Sep 17 21:18:55.445523 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:55.445494    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/kserve-container/0.log"
Sep 17 21:18:55.468034 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:55.468008    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/kube-rbac-proxy/0.log"
Sep 17 21:18:55.486868 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:55.486846    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/agent/0.log"
Sep 17 21:18:55.511424 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:55.511401    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/storage-initializer/0.log"
Sep 17 21:18:56.399913 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:56.399889    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/kserve-container/0.log"
Sep 17 21:18:56.413430 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:56.413411    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/kube-rbac-proxy/0.log"
Sep 17 21:18:56.425511 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:56.425494    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/agent/0.log"
Sep 17 21:18:56.436968 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:56.436948    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/storage-initializer/0.log"
Sep 17 21:18:57.256381 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:57.256355    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/kserve-container/0.log"
Sep 17 21:18:57.267590 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:57.267559    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/kube-rbac-proxy/0.log"
Sep 17 21:18:57.278129 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:57.278113    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/agent/0.log"
Sep 17 21:18:57.289045 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:57.289027    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/storage-initializer/0.log"
Sep 17 21:18:58.125985 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:58.125957    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/kserve-container/0.log"
Sep 17 21:18:58.149743 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:58.149718    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/kube-rbac-proxy/0.log"
Sep 17 21:18:58.160703 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:58.160684    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/agent/0.log"
Sep 17 21:18:58.178577 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:58.178559    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/storage-initializer/0.log"
Sep 17 21:18:59.005374 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:59.005350    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/kserve-container/0.log"
Sep 17 21:18:59.018557 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:59.018539    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/kube-rbac-proxy/0.log"
Sep 17 21:18:59.029637 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:59.029619    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/agent/0.log"
Sep 17 21:18:59.041551 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:59.041535    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/storage-initializer/0.log"
Sep 17 21:18:59.220723 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:59.220688    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 17 21:18:59.881546 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:59.881522    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/kserve-container/0.log"
Sep 17 21:18:59.906840 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:59.906819    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/kube-rbac-proxy/0.log"
Sep 17 21:18:59.927085 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:59.927068    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/agent/0.log"
Sep 17 21:18:59.942111 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:18:59.942094    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/storage-initializer/0.log"
Sep 17 21:19:00.785780 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:00.785749    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/kserve-container/0.log"
Sep 17 21:19:00.797952 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:00.797931    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/kube-rbac-proxy/0.log"
Sep 17 21:19:00.808458 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:00.808436    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/agent/0.log"
Sep 17 21:19:00.820475 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:00.820454    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/storage-initializer/0.log"
Sep 17 21:19:01.455364 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:01.455312    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 21:19:01.455567 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:01.455449    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"
Sep 17 21:19:01.455671 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:01.455654    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:19:01.455739 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:01.455728    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"
Sep 17 21:19:01.627346 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:01.627322    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/kserve-container/0.log"
Sep 17 21:19:01.643296 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:01.643276    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/kube-rbac-proxy/0.log"
Sep 17 21:19:01.663182 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:01.663158    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/agent/0.log"
Sep 17 21:19:01.675022 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:01.675005    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/storage-initializer/0.log"
Sep 17 21:19:02.501254 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:02.501225    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/kserve-container/0.log"
Sep 17 21:19:02.512283 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:02.512258    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/kube-rbac-proxy/0.log"
Sep 17 21:19:02.522883 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:02.522866    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/agent/0.log"
Sep 17 21:19:02.533899 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:02.533881    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/storage-initializer/0.log"
Sep 17 21:19:03.363836 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:03.363764    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/kserve-container/0.log"
Sep 17 21:19:03.375629 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:03.375597    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/kube-rbac-proxy/0.log"
Sep 17 21:19:03.386636 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:03.386617    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/agent/0.log"
Sep 17 21:19:03.397558 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:03.397541    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/storage-initializer/0.log"
Sep 17 21:19:04.220009 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:04.219970    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 17 21:19:04.239552 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:04.239515    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/kserve-container/0.log"
Sep 17 21:19:04.251270 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:04.251249    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/kube-rbac-proxy/0.log"
Sep 17 21:19:04.261052 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:04.261035    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/agent/0.log"
Sep 17 21:19:04.270034 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:04.270017    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/storage-initializer/0.log"
Sep 17 21:19:05.151494 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:05.151472    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/kserve-container/0.log"
Sep 17 21:19:05.177936 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:05.177913    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/kube-rbac-proxy/0.log"
Sep 17 21:19:05.206871 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:05.206853    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/agent/0.log"
Sep 17 21:19:05.238917 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:05.238901    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/storage-initializer/0.log"
Sep 17 21:19:06.142602 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:06.142577    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/kserve-container/0.log"
Sep 17 21:19:06.154935 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:06.154915    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/kube-rbac-proxy/0.log"
Sep 17 21:19:06.172158 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:06.172137    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/agent/0.log"
Sep 17 21:19:06.184875 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:06.184856    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-797ff577-vsv2w_a68d8b52-888b-4113-972c-3fc6aa9c7dc8/storage-initializer/0.log"
Sep 17 21:19:07.062753 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.062730    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"
Sep 17 21:19:07.205137 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.205053    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a68d8b52-888b-4113-972c-3fc6aa9c7dc8-kserve-provision-location\") pod \"a68d8b52-888b-4113-972c-3fc6aa9c7dc8\" (UID: \"a68d8b52-888b-4113-972c-3fc6aa9c7dc8\") "
Sep 17 21:19:07.205137 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.205092    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a68d8b52-888b-4113-972c-3fc6aa9c7dc8-kube-rbac-proxy-sar-config\") pod \"a68d8b52-888b-4113-972c-3fc6aa9c7dc8\" (UID: \"a68d8b52-888b-4113-972c-3fc6aa9c7dc8\") "
Sep 17 21:19:07.205137 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.205133    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a68d8b52-888b-4113-972c-3fc6aa9c7dc8-proxy-tls\") pod \"a68d8b52-888b-4113-972c-3fc6aa9c7dc8\" (UID: \"a68d8b52-888b-4113-972c-3fc6aa9c7dc8\") "
Sep 17 21:19:07.205452 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.205173    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7t46\" (UniqueName: \"kubernetes.io/projected/a68d8b52-888b-4113-972c-3fc6aa9c7dc8-kube-api-access-x7t46\") pod \"a68d8b52-888b-4113-972c-3fc6aa9c7dc8\" (UID: \"a68d8b52-888b-4113-972c-3fc6aa9c7dc8\") "
Sep 17 21:19:07.205452 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.205413    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a68d8b52-888b-4113-972c-3fc6aa9c7dc8-cabundle-cert\") pod \"a68d8b52-888b-4113-972c-3fc6aa9c7dc8\" (UID: \"a68d8b52-888b-4113-972c-3fc6aa9c7dc8\") "
Sep 17 21:19:07.205560 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.205441    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68d8b52-888b-4113-972c-3fc6aa9c7dc8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a68d8b52-888b-4113-972c-3fc6aa9c7dc8" (UID: "a68d8b52-888b-4113-972c-3fc6aa9c7dc8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:19:07.205560 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.205504    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a68d8b52-888b-4113-972c-3fc6aa9c7dc8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a68d8b52-888b-4113-972c-3fc6aa9c7dc8" (UID: "a68d8b52-888b-4113-972c-3fc6aa9c7dc8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:19:07.205693 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.205674    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a68d8b52-888b-4113-972c-3fc6aa9c7dc8-kserve-provision-location\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:19:07.205751 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.205695    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a68d8b52-888b-4113-972c-3fc6aa9c7dc8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:19:07.205751 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.205725    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a68d8b52-888b-4113-972c-3fc6aa9c7dc8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a68d8b52-888b-4113-972c-3fc6aa9c7dc8" (UID: "a68d8b52-888b-4113-972c-3fc6aa9c7dc8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:19:07.207282 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.207256    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a68d8b52-888b-4113-972c-3fc6aa9c7dc8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a68d8b52-888b-4113-972c-3fc6aa9c7dc8" (UID: "a68d8b52-888b-4113-972c-3fc6aa9c7dc8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:19:07.207377 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.207319    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a68d8b52-888b-4113-972c-3fc6aa9c7dc8-kube-api-access-x7t46" (OuterVolumeSpecName: "kube-api-access-x7t46") pod "a68d8b52-888b-4113-972c-3fc6aa9c7dc8" (UID: "a68d8b52-888b-4113-972c-3fc6aa9c7dc8"). InnerVolumeSpecName "kube-api-access-x7t46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:19:07.306411 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.306386    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a68d8b52-888b-4113-972c-3fc6aa9c7dc8-cabundle-cert\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:19:07.306411 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.306406    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a68d8b52-888b-4113-972c-3fc6aa9c7dc8-proxy-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:19:07.306549 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.306415    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7t46\" (UniqueName: \"kubernetes.io/projected/a68d8b52-888b-4113-972c-3fc6aa9c7dc8-kube-api-access-x7t46\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Sep 17 21:19:07.677983 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.677948    2566 generic.go:356] "Generic (PLEG): container finished" podID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerID="326e1261ecf60dd08ae23a45e9e79db762bdbda8495acf82e6d9aecf6c8277a4" exitCode=0
Sep 17 21:19:07.678152 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.678011    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"
Sep 17 21:19:07.678152 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.678029    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" event={"ID":"a68d8b52-888b-4113-972c-3fc6aa9c7dc8","Type":"ContainerDied","Data":"326e1261ecf60dd08ae23a45e9e79db762bdbda8495acf82e6d9aecf6c8277a4"}
Sep 17 21:19:07.678152 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.678107    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w" event={"ID":"a68d8b52-888b-4113-972c-3fc6aa9c7dc8","Type":"ContainerDied","Data":"de25776f20847747c2adbf98a45324b9d1fd2907cc0fcd7547eba3e227111271"}
Sep 17 21:19:07.678152 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.678150    2566 scope.go:117] "RemoveContainer" containerID="326e1261ecf60dd08ae23a45e9e79db762bdbda8495acf82e6d9aecf6c8277a4"
Sep 17 21:19:07.686654 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.686638    2566 scope.go:117] "RemoveContainer" containerID="eb46fde6db72c635b10808624c17800ee4518ab1e1ffacb5ef68dbf73cf4f5d0"
Sep 17 21:19:07.693863 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.693849    2566 scope.go:117] "RemoveContainer" containerID="89305c22b136737018182cf252d0a32f539048e0a062cc5f0a40f62a15920fa0"
Sep 17 21:19:07.700493 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.700476    2566 scope.go:117] "RemoveContainer" containerID="3386af51d158ffe757dc38edb6e424ce92a667eab57d24d070be4c48c75a2333"
Sep 17 21:19:07.705337 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.705316    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"]
Sep 17 21:19:07.708270 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.708245    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-797ff577-vsv2w"]
Sep 17 21:19:07.708976 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.708959    2566 scope.go:117] "RemoveContainer" containerID="326e1261ecf60dd08ae23a45e9e79db762bdbda8495acf82e6d9aecf6c8277a4"
Sep 17 21:19:07.709280 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:19:07.709262    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"326e1261ecf60dd08ae23a45e9e79db762bdbda8495acf82e6d9aecf6c8277a4\": container with ID starting with 326e1261ecf60dd08ae23a45e9e79db762bdbda8495acf82e6d9aecf6c8277a4 not found: ID does not exist" containerID="326e1261ecf60dd08ae23a45e9e79db762bdbda8495acf82e6d9aecf6c8277a4"
Sep 17 21:19:07.709344 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.709288    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"326e1261ecf60dd08ae23a45e9e79db762bdbda8495acf82e6d9aecf6c8277a4"} err="failed to get container status \"326e1261ecf60dd08ae23a45e9e79db762bdbda8495acf82e6d9aecf6c8277a4\": rpc error: code = NotFound desc = could not find container \"326e1261ecf60dd08ae23a45e9e79db762bdbda8495acf82e6d9aecf6c8277a4\": container with ID starting with 326e1261ecf60dd08ae23a45e9e79db762bdbda8495acf82e6d9aecf6c8277a4 not found: ID does not exist"
Sep 17 21:19:07.709344 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.709306    2566 scope.go:117] "RemoveContainer" containerID="eb46fde6db72c635b10808624c17800ee4518ab1e1ffacb5ef68dbf73cf4f5d0"
Sep 17 21:19:07.709559 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:19:07.709543    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb46fde6db72c635b10808624c17800ee4518ab1e1ffacb5ef68dbf73cf4f5d0\": container with ID starting with eb46fde6db72c635b10808624c17800ee4518ab1e1ffacb5ef68dbf73cf4f5d0 not found: ID does not exist" containerID="eb46fde6db72c635b10808624c17800ee4518ab1e1ffacb5ef68dbf73cf4f5d0"
Sep 17 21:19:07.709602 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.709571    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb46fde6db72c635b10808624c17800ee4518ab1e1ffacb5ef68dbf73cf4f5d0"} err="failed to get container status \"eb46fde6db72c635b10808624c17800ee4518ab1e1ffacb5ef68dbf73cf4f5d0\": rpc error: code = NotFound desc = could not find container \"eb46fde6db72c635b10808624c17800ee4518ab1e1ffacb5ef68dbf73cf4f5d0\": container with ID starting with eb46fde6db72c635b10808624c17800ee4518ab1e1ffacb5ef68dbf73cf4f5d0 not found: ID does not exist"
Sep 17 21:19:07.709602 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.709587    2566 scope.go:117] "RemoveContainer" containerID="89305c22b136737018182cf252d0a32f539048e0a062cc5f0a40f62a15920fa0"
Sep 17 21:19:07.709821 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:19:07.709800    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89305c22b136737018182cf252d0a32f539048e0a062cc5f0a40f62a15920fa0\": container with ID starting with 89305c22b136737018182cf252d0a32f539048e0a062cc5f0a40f62a15920fa0 not found: ID does not exist" containerID="89305c22b136737018182cf252d0a32f539048e0a062cc5f0a40f62a15920fa0"
Sep 17 21:19:07.709865 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.709827    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89305c22b136737018182cf252d0a32f539048e0a062cc5f0a40f62a15920fa0"} err="failed to get container status \"89305c22b136737018182cf252d0a32f539048e0a062cc5f0a40f62a15920fa0\": rpc error: code = NotFound desc = could not find container \"89305c22b136737018182cf252d0a32f539048e0a062cc5f0a40f62a15920fa0\": container with ID starting with 89305c22b136737018182cf252d0a32f539048e0a062cc5f0a40f62a15920fa0 not found: ID does not exist"
Sep 17 21:19:07.709865 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.709841    2566 scope.go:117] "RemoveContainer" containerID="3386af51d158ffe757dc38edb6e424ce92a667eab57d24d070be4c48c75a2333"
Sep 17 21:19:07.710061 ip-10-0-138-140 kubenswrapper[2566]: E0917 21:19:07.710044    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3386af51d158ffe757dc38edb6e424ce92a667eab57d24d070be4c48c75a2333\": container with ID starting with 3386af51d158ffe757dc38edb6e424ce92a667eab57d24d070be4c48c75a2333 not found: ID does not exist" containerID="3386af51d158ffe757dc38edb6e424ce92a667eab57d24d070be4c48c75a2333"
Sep 17 21:19:07.710099 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:07.710065    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3386af51d158ffe757dc38edb6e424ce92a667eab57d24d070be4c48c75a2333"} err="failed to get container status \"3386af51d158ffe757dc38edb6e424ce92a667eab57d24d070be4c48c75a2333\": rpc error: code = NotFound desc = could not find container \"3386af51d158ffe757dc38edb6e424ce92a667eab57d24d070be4c48c75a2333\": container with ID starting with 3386af51d158ffe757dc38edb6e424ce92a667eab57d24d070be4c48c75a2333 not found: ID does not exist"
Sep 17 21:19:08.459279 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:08.459246    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" path="/var/lib/kubelet/pods/a68d8b52-888b-4113-972c-3fc6aa9c7dc8/volumes"
Sep 17 21:19:08.593564 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:08.593535    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zzvpz/must-gather-xqrh7"]
Sep 17 21:19:08.593806 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:08.593795    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="kube-rbac-proxy"
Sep 17 21:19:08.593849 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:08.593808    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="kube-rbac-proxy"
Sep 17 21:19:08.593849 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:08.593816    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="agent"
Sep 17 21:19:08.593849 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:08.593822    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="agent"
Sep 17 21:19:08.593849 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:08.593835    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="storage-initializer"
Sep 17 21:19:08.593849 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:08.593841    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="storage-initializer"
Sep 17 21:19:08.593849 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:08.593849    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="kserve-container"
Sep 17 21:19:08.594029 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:08.593856    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="kserve-container"
Sep 17 21:19:08.594029 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:08.593905    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="agent"
Sep 17 21:19:08.594029 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:08.593913    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="kserve-container"
Sep 17 21:19:08.594029 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:08.593922    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a68d8b52-888b-4113-972c-3fc6aa9c7dc8" containerName="kube-rbac-proxy"
Sep 17 21:19:08.598665 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:08.598650    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zzvpz/must-gather-xqrh7"
Sep 17 21:19:08.602131 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:08.602112    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zzvpz\"/\"kube-root-ca.crt\""
Sep 17 21:19:08.602274 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:08.602262    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zzvpz\"/\"default-dockercfg-g2pnc\""
Sep 17 21:19:08.602592 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:08.602572    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zzvpz\"/\"openshift-service-ca.crt\""
Sep 17 21:19:08.612814 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:08.612794    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zzvpz/must-gather-xqrh7"]
Sep 17 21:19:08.715540 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:08.715516    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7ea1deb7-1711-4574-87bf-95f9c50c306f-must-gather-output\") pod \"must-gather-xqrh7\" (UID: \"7ea1deb7-1711-4574-87bf-95f9c50c306f\") " pod="openshift-must-gather-zzvpz/must-gather-xqrh7"
Sep 17 21:19:08.715693 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:08.715609    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glm2v\" (UniqueName: \"kubernetes.io/projected/7ea1deb7-1711-4574-87bf-95f9c50c306f-kube-api-access-glm2v\") pod \"must-gather-xqrh7\" (UID: \"7ea1deb7-1711-4574-87bf-95f9c50c306f\") " pod="openshift-must-gather-zzvpz/must-gather-xqrh7"
Sep 17 21:19:08.817000 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:08.816963    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7ea1deb7-1711-4574-87bf-95f9c50c306f-must-gather-output\") pod \"must-gather-xqrh7\" (UID: \"7ea1deb7-1711-4574-87bf-95f9c50c306f\") " pod="openshift-must-gather-zzvpz/must-gather-xqrh7"
Sep 17 21:19:08.817193 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:08.817045    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glm2v\" (UniqueName: \"kubernetes.io/projected/7ea1deb7-1711-4574-87bf-95f9c50c306f-kube-api-access-glm2v\") pod \"must-gather-xqrh7\" (UID: \"7ea1deb7-1711-4574-87bf-95f9c50c306f\") " pod="openshift-must-gather-zzvpz/must-gather-xqrh7"
Sep 17 21:19:08.817319 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:08.817298    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7ea1deb7-1711-4574-87bf-95f9c50c306f-must-gather-output\") pod \"must-gather-xqrh7\" (UID: \"7ea1deb7-1711-4574-87bf-95f9c50c306f\") " pod="openshift-must-gather-zzvpz/must-gather-xqrh7"
Sep 17 21:19:08.825300 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:08.825268    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glm2v\" (UniqueName: \"kubernetes.io/projected/7ea1deb7-1711-4574-87bf-95f9c50c306f-kube-api-access-glm2v\") pod \"must-gather-xqrh7\" (UID: \"7ea1deb7-1711-4574-87bf-95f9c50c306f\") " pod="openshift-must-gather-zzvpz/must-gather-xqrh7"
Sep 17 21:19:08.907422 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:08.907391    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zzvpz/must-gather-xqrh7"
Sep 17 21:19:09.174923 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:09.174845    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zzvpz/must-gather-xqrh7"]
Sep 17 21:19:09.177893 ip-10-0-138-140 kubenswrapper[2566]: W0917 21:19:09.177867    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ea1deb7_1711_4574_87bf_95f9c50c306f.slice/crio-9d6fba7e295553c8a92a358d40efb6b537791ab1d302d12f46f813fbd5fca209 WatchSource:0}: Error finding container 9d6fba7e295553c8a92a358d40efb6b537791ab1d302d12f46f813fbd5fca209: Status 404 returned error can't find the container with id 9d6fba7e295553c8a92a358d40efb6b537791ab1d302d12f46f813fbd5fca209
Sep 17 21:19:09.180060 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:09.180045    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:19:09.688143 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:09.688110    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zzvpz/must-gather-xqrh7" event={"ID":"7ea1deb7-1711-4574-87bf-95f9c50c306f","Type":"ContainerStarted","Data":"9d6fba7e295553c8a92a358d40efb6b537791ab1d302d12f46f813fbd5fca209"}
Sep 17 21:19:10.697172 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:10.697135    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zzvpz/must-gather-xqrh7" event={"ID":"7ea1deb7-1711-4574-87bf-95f9c50c306f","Type":"ContainerStarted","Data":"db3d22d40d3f6b98ffcb62ff1cb0be676c9ceda3651367f1bfebb8a11c9de4d7"}
Sep 17 21:19:10.698499 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:10.697721    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zzvpz/must-gather-xqrh7" event={"ID":"7ea1deb7-1711-4574-87bf-95f9c50c306f","Type":"ContainerStarted","Data":"e4f928f2cf06cdc4890ed69623c5bb30a3f423140ca045e424c67f5f0473f8f8"}
Sep 17 21:19:11.519679 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:11.519649    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ltk4w_3aa9cc3a-9ccd-4894-b3ad-14ad007ced8a/global-pull-secret-syncer/0.log"
Sep 17 21:19:11.628658 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:11.628591    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6mj5z_582bf447-0797-4411-bce3-89e1c2455f6d/konnectivity-agent/0.log"
Sep 17 21:19:11.724079 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:11.724044    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-140.ec2.internal_e97c66319ae520682604e45d82c8b847/haproxy/0.log"
Sep 17 21:19:15.086454 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:15.086405    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bf415666-9904-4e67-a1cd-2ad770aa292f/alertmanager/0.log"
Sep 17 21:19:15.113882 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:15.113857    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bf415666-9904-4e67-a1cd-2ad770aa292f/config-reloader/0.log"
Sep 17 21:19:15.140246 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:15.140199    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bf415666-9904-4e67-a1cd-2ad770aa292f/kube-rbac-proxy-web/0.log"
Sep 17 21:19:15.167686 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:15.167633    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bf415666-9904-4e67-a1cd-2ad770aa292f/kube-rbac-proxy/0.log"
Sep 17 21:19:15.208205 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:15.208124    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bf415666-9904-4e67-a1cd-2ad770aa292f/kube-rbac-proxy-metric/0.log"
Sep 17 21:19:15.243138 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:15.243061    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bf415666-9904-4e67-a1cd-2ad770aa292f/prom-label-proxy/0.log"
Sep 17 21:19:15.268686 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:15.268659    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bf415666-9904-4e67-a1cd-2ad770aa292f/init-config-reloader/0.log"
Sep 17 21:19:15.349110 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:15.349003    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-brjxp_dac1817a-9871-4cb6-848f-56465179a242/kube-state-metrics/0.log"
Sep 17 21:19:15.375280 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:15.375238    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-brjxp_dac1817a-9871-4cb6-848f-56465179a242/kube-rbac-proxy-main/0.log"
Sep 17 21:19:15.400881 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:15.400850    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-brjxp_dac1817a-9871-4cb6-848f-56465179a242/kube-rbac-proxy-self/0.log"
Sep 17 21:19:15.530441 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:15.530392    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-77pdr_e8fa9f0a-d3a0-47b1-943b-3f6c548f4290/node-exporter/0.log"
Sep 17 21:19:15.575535 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:15.575503    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-77pdr_e8fa9f0a-d3a0-47b1-943b-3f6c548f4290/kube-rbac-proxy/0.log"
Sep 17 21:19:15.620321 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:15.620244    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-77pdr_e8fa9f0a-d3a0-47b1-943b-3f6c548f4290/init-textfile/0.log"
Sep 17 21:19:16.568538 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:16.568513    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55c8ddbbd9-2w5f2_566fd44b-5ea9-46f4-bbd1-86ed826cad04/telemeter-client/0.log"
Sep 17 21:19:16.593139 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:16.593085    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55c8ddbbd9-2w5f2_566fd44b-5ea9-46f4-bbd1-86ed826cad04/reload/0.log"
Sep 17 21:19:16.620084 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:16.620058    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55c8ddbbd9-2w5f2_566fd44b-5ea9-46f4-bbd1-86ed826cad04/kube-rbac-proxy/0.log"
Sep 17 21:19:18.757279 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:18.757247    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t2rlv_ed6472a6-7979-43b0-a549-7ca5c5d8fb0f/console-operator/1.log"
Sep 17 21:19:18.762874 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:18.762853    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t2rlv_ed6472a6-7979-43b0-a549-7ca5c5d8fb0f/console-operator/2.log"
Sep 17 21:19:19.534299 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:19.534239    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zzvpz/must-gather-xqrh7" podStartSLOduration=10.731648408 podStartE2EDuration="11.534197763s" totalImagesPullingTime="802.549355ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:19:10.735303928 +0000 UTC m=+1792.346037154" watchObservedRunningTime="2026-09-17 21:19:19.534197763 +0000 UTC m=+1801.144931054"
Sep 17 21:19:19.534560 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:19.534494    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zzvpz/perf-node-gather-daemonset-f5m8m"]
Sep 17 21:19:19.540283 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:19.540258    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zzvpz/perf-node-gather-daemonset-f5m8m"
Sep 17 21:19:19.547740 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:19.547716    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zzvpz/perf-node-gather-daemonset-f5m8m"]
Sep 17 21:19:19.714991 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:19.714959    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3d3ae5ae-44af-4857-baea-191443bbe36c-proc\") pod \"perf-node-gather-daemonset-f5m8m\" (UID: \"3d3ae5ae-44af-4857-baea-191443bbe36c\") " pod="openshift-must-gather-zzvpz/perf-node-gather-daemonset-f5m8m"
Sep 17 21:19:19.715159 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:19.715006    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njsn2\" (UniqueName: \"kubernetes.io/projected/3d3ae5ae-44af-4857-baea-191443bbe36c-kube-api-access-njsn2\") pod \"perf-node-gather-daemonset-f5m8m\" (UID: \"3d3ae5ae-44af-4857-baea-191443bbe36c\") " pod="openshift-must-gather-zzvpz/perf-node-gather-daemonset-f5m8m"
Sep 17 21:19:19.715159 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:19.715028    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d3ae5ae-44af-4857-baea-191443bbe36c-lib-modules\") pod \"perf-node-gather-daemonset-f5m8m\" (UID: \"3d3ae5ae-44af-4857-baea-191443bbe36c\") " pod="openshift-must-gather-zzvpz/perf-node-gather-daemonset-f5m8m"
Sep 17 21:19:19.715159 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:19.715133    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3d3ae5ae-44af-4857-baea-191443bbe36c-podres\") pod \"perf-node-gather-daemonset-f5m8m\" (UID: \"3d3ae5ae-44af-4857-baea-191443bbe36c\") " pod="openshift-must-gather-zzvpz/perf-node-gather-daemonset-f5m8m"
Sep 17 21:19:19.715304 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:19.715163    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d3ae5ae-44af-4857-baea-191443bbe36c-sys\") pod \"perf-node-gather-daemonset-f5m8m\" (UID: \"3d3ae5ae-44af-4857-baea-191443bbe36c\") " pod="openshift-must-gather-zzvpz/perf-node-gather-daemonset-f5m8m"
Sep 17 21:19:19.716854 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:19.716833    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-lk69f_d19d48a3-03c2-4fe4-a052-8b7e87b0055a/volume-data-source-validator/0.log"
Sep 17 21:19:19.816648 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:19.816561    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d3ae5ae-44af-4857-baea-191443bbe36c-lib-modules\") pod \"perf-node-gather-daemonset-f5m8m\" (UID: \"3d3ae5ae-44af-4857-baea-191443bbe36c\") " pod="openshift-must-gather-zzvpz/perf-node-gather-daemonset-f5m8m"
Sep 17 21:19:19.816648 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:19.816647    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3d3ae5ae-44af-4857-baea-191443bbe36c-podres\") pod \"perf-node-gather-daemonset-f5m8m\" (UID: \"3d3ae5ae-44af-4857-baea-191443bbe36c\") " pod="openshift-must-gather-zzvpz/perf-node-gather-daemonset-f5m8m"
Sep 17 21:19:19.817145 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:19.816677    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d3ae5ae-44af-4857-baea-191443bbe36c-sys\") pod \"perf-node-gather-daemonset-f5m8m\" (UID: \"3d3ae5ae-44af-4857-baea-191443bbe36c\") " pod="openshift-must-gather-zzvpz/perf-node-gather-daemonset-f5m8m"
Sep 17 21:19:19.817145 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:19.816724    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3d3ae5ae-44af-4857-baea-191443bbe36c-proc\") pod \"perf-node-gather-daemonset-f5m8m\" (UID: \"3d3ae5ae-44af-4857-baea-191443bbe36c\") " pod="openshift-must-gather-zzvpz/perf-node-gather-daemonset-f5m8m"
Sep 17 21:19:19.817145 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:19.816775    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njsn2\" (UniqueName: \"kubernetes.io/projected/3d3ae5ae-44af-4857-baea-191443bbe36c-kube-api-access-njsn2\") pod \"perf-node-gather-daemonset-f5m8m\" (UID: \"3d3ae5ae-44af-4857-baea-191443bbe36c\") " pod="openshift-must-gather-zzvpz/perf-node-gather-daemonset-f5m8m"
Sep 17 21:19:19.817145 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:19.817123    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d3ae5ae-44af-4857-baea-191443bbe36c-sys\") pod \"perf-node-gather-daemonset-f5m8m\" (UID: \"3d3ae5ae-44af-4857-baea-191443bbe36c\") " pod="openshift-must-gather-zzvpz/perf-node-gather-daemonset-f5m8m"
Sep 17 21:19:19.817145 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:19.817136    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3d3ae5ae-44af-4857-baea-191443bbe36c-proc\") pod \"perf-node-gather-daemonset-f5m8m\" (UID: \"3d3ae5ae-44af-4857-baea-191443bbe36c\") " pod="openshift-must-gather-zzvpz/perf-node-gather-daemonset-f5m8m"
Sep 17 21:19:19.817420 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:19.817181    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3d3ae5ae-44af-4857-baea-191443bbe36c-podres\") pod \"perf-node-gather-daemonset-f5m8m\" (UID: \"3d3ae5ae-44af-4857-baea-191443bbe36c\") " pod="openshift-must-gather-zzvpz/perf-node-gather-daemonset-f5m8m"
Sep 17 21:19:19.817420 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:19.817270    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d3ae5ae-44af-4857-baea-191443bbe36c-lib-modules\") pod \"perf-node-gather-daemonset-f5m8m\" (UID: \"3d3ae5ae-44af-4857-baea-191443bbe36c\") " pod="openshift-must-gather-zzvpz/perf-node-gather-daemonset-f5m8m"
Sep 17 21:19:19.825749 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:19.825729    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njsn2\" (UniqueName: \"kubernetes.io/projected/3d3ae5ae-44af-4857-baea-191443bbe36c-kube-api-access-njsn2\") pod \"perf-node-gather-daemonset-f5m8m\" (UID: \"3d3ae5ae-44af-4857-baea-191443bbe36c\") " pod="openshift-must-gather-zzvpz/perf-node-gather-daemonset-f5m8m"
Sep 17 21:19:19.853405 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:19.853383    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zzvpz/perf-node-gather-daemonset-f5m8m"
Sep 17 21:19:20.144389 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:20.142305    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zzvpz/perf-node-gather-daemonset-f5m8m"]
Sep 17 21:19:20.145742 ip-10-0-138-140 kubenswrapper[2566]: W0917 21:19:20.145716    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3d3ae5ae_44af_4857_baea_191443bbe36c.slice/crio-cdb9689c3be403011fa5bf2254ab0d82f3ac35ac7d872700a9102f18fb4011b7 WatchSource:0}: Error finding container cdb9689c3be403011fa5bf2254ab0d82f3ac35ac7d872700a9102f18fb4011b7: Status 404 returned error can't find the container with id cdb9689c3be403011fa5bf2254ab0d82f3ac35ac7d872700a9102f18fb4011b7
Sep 17 21:19:20.510384 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:20.510352    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t2rlv_ed6472a6-7979-43b0-a549-7ca5c5d8fb0f/console-operator/1.log"
Sep 17 21:19:20.510573 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:20.510446    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t2rlv_ed6472a6-7979-43b0-a549-7ca5c5d8fb0f/console-operator/1.log"
Sep 17 21:19:20.518992 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:20.518971    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnqx7_402f55a4-24b6-47c6-b719-9c6a5b473434/ovn-acl-logging/0.log"
Sep 17 21:19:20.519240 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:20.519199    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnqx7_402f55a4-24b6-47c6-b719-9c6a5b473434/ovn-acl-logging/0.log"
Sep 17 21:19:20.598747 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:20.598720    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-78ncn_ae0c2874-4c74-491b-8fa4-4ff0fcc84c00/dns/0.log"
Sep 17 21:19:20.624032 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:20.624008    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-78ncn_ae0c2874-4c74-491b-8fa4-4ff0fcc84c00/kube-rbac-proxy/0.log"
Sep 17 21:19:20.737786 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:20.737758    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j7t6x_5f5d4daa-933c-47d6-b93a-6f83ed7ab23f/dns-node-resolver/0.log"
Sep 17 21:19:20.756983 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:20.756952    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zzvpz/perf-node-gather-daemonset-f5m8m" event={"ID":"3d3ae5ae-44af-4857-baea-191443bbe36c","Type":"ContainerStarted","Data":"44f1ed685b3c4ff5f370f14638441bbc5dbf2d0c9fb10e14cd0e1e959c453fc0"}
Sep 17 21:19:20.757161 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:20.757004    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zzvpz/perf-node-gather-daemonset-f5m8m" event={"ID":"3d3ae5ae-44af-4857-baea-191443bbe36c","Type":"ContainerStarted","Data":"cdb9689c3be403011fa5bf2254ab0d82f3ac35ac7d872700a9102f18fb4011b7"}
Sep 17 21:19:20.757161 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:20.757038    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zzvpz/perf-node-gather-daemonset-f5m8m"
Sep 17 21:19:20.776144 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:20.776031    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zzvpz/perf-node-gather-daemonset-f5m8m" podStartSLOduration=1.776014336 podStartE2EDuration="1.776014336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:19:20.773685081 +0000 UTC m=+1802.384418311" watchObservedRunningTime="2026-09-17 21:19:20.776014336 +0000 UTC m=+1802.386747567"
Sep 17 21:19:21.264527 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:21.264501    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j8n57_5e2bd987-6d8a-457b-ad67-f1e6597864e5/node-ca/0.log"
Sep 17 21:19:22.405868 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:22.405847    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vnd5t_850f96d3-cdee-4f2e-8e36-f32466aa1bbb/serve-healthcheck-canary/0.log"
Sep 17 21:19:22.832599 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:22.832563    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7m89j_102ec935-a1f0-4e2b-84ae-8832e11a1033/kube-rbac-proxy/0.log"
Sep 17 21:19:22.852365 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:22.852339    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7m89j_102ec935-a1f0-4e2b-84ae-8832e11a1033/exporter/0.log"
Sep 17 21:19:22.871979 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:22.871931    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7m89j_102ec935-a1f0-4e2b-84ae-8832e11a1033/extractor/0.log"
Sep 17 21:19:25.494490 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:25.494466    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-gcv9m_0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9/s3-init/0.log"
Sep 17 21:19:25.511017 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:25.510999    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-gcv9m_0ff37ebc-9a3c-4c6d-8d33-a6b6bb7863d9/download-hf-model/0.log"
Sep 17 21:19:26.770619 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:26.770548    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zzvpz/perf-node-gather-daemonset-f5m8m"
Sep 17 21:19:29.769718 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:29.769688    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-7mfg4_b92f4da1-f4c2-428b-ad13-8892f296be13/migrator/0.log"
Sep 17 21:19:29.790880 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:29.790859    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-7mfg4_b92f4da1-f4c2-428b-ad13-8892f296be13/graceful-termination/0.log"
Sep 17 21:19:31.364965 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:31.364895    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rw72g_67a00ce8-eae6-48cb-896a-bb6de23904e9/kube-multus-additional-cni-plugins/0.log"
Sep 17 21:19:31.385461 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:31.385442    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rw72g_67a00ce8-eae6-48cb-896a-bb6de23904e9/egress-router-binary-copy/0.log"
Sep 17 21:19:31.405074 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:31.405052    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rw72g_67a00ce8-eae6-48cb-896a-bb6de23904e9/cni-plugins/0.log"
Sep 17 21:19:31.426506 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:31.426478    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rw72g_67a00ce8-eae6-48cb-896a-bb6de23904e9/bond-cni-plugin/0.log"
Sep 17 21:19:31.446898 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:31.446878    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rw72g_67a00ce8-eae6-48cb-896a-bb6de23904e9/routeoverride-cni/0.log"
Sep 17 21:19:31.470196 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:31.470181    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rw72g_67a00ce8-eae6-48cb-896a-bb6de23904e9/whereabouts-cni-bincopy/0.log"
Sep 17 21:19:31.492249 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:31.492228    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rw72g_67a00ce8-eae6-48cb-896a-bb6de23904e9/whereabouts-cni/0.log"
Sep 17 21:19:31.722204 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:31.722180    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-h5f4t_5caabfba-6743-414e-8d96-caf028817a3b/kube-multus/0.log"
Sep 17 21:19:31.901852 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:31.901826    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-sknnf_8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7/network-metrics-daemon/0.log"
Sep 17 21:19:31.920703 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:31.920680    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-sknnf_8d4efba7-875e-4c04-9ebe-2bd9e0e0dfa7/kube-rbac-proxy/0.log"
Sep 17 21:19:32.890481 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:32.890450    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnqx7_402f55a4-24b6-47c6-b719-9c6a5b473434/ovn-controller/0.log"
Sep 17 21:19:32.906021 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:32.906001    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnqx7_402f55a4-24b6-47c6-b719-9c6a5b473434/ovn-acl-logging/1.log"
Sep 17 21:19:32.906151 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:32.906099    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnqx7_402f55a4-24b6-47c6-b719-9c6a5b473434/ovn-acl-logging/0.log"
Sep 17 21:19:32.931115 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:32.931092    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnqx7_402f55a4-24b6-47c6-b719-9c6a5b473434/kube-rbac-proxy-node/0.log"
Sep 17 21:19:32.953823 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:32.953803    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnqx7_402f55a4-24b6-47c6-b719-9c6a5b473434/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 21:19:32.971667 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:32.971650    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnqx7_402f55a4-24b6-47c6-b719-9c6a5b473434/northd/0.log"
Sep 17 21:19:32.993406 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:32.993374    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnqx7_402f55a4-24b6-47c6-b719-9c6a5b473434/nbdb/0.log"
Sep 17 21:19:33.013104 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:33.013082    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnqx7_402f55a4-24b6-47c6-b719-9c6a5b473434/sbdb/0.log"
Sep 17 21:19:33.056264 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:33.056249    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnqx7_402f55a4-24b6-47c6-b719-9c6a5b473434/ovnkube-controller/0.log"
Sep 17 21:19:34.221538 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:34.221513    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-9tkmw_3825a8f7-bf7a-4e42-8fd3-80ec1bfc615b/network-check-target-container/0.log"
Sep 17 21:19:35.194605 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:35.194578    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-4nh74_21362652-a68e-4d69-8cea-69002798058b/iptables-alerter/0.log"
Sep 17 21:19:36.025304 ip-10-0-138-140 kubenswrapper[2566]: I0917 21:19:36.025277    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-bzvzr_ce8d6a4d-e013-4e57-a540-14ddc06d7745/tuned/0.log"