Sep 17 17:56:56.523414 ip-10-0-134-209 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 17:56:59.828191 ip-10-0-134-209 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:56:59.828191 ip-10-0-134-209 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 17:56:59.828191 ip-10-0-134-209 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:56:59.828191 ip-10-0-134-209 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 17:56:59.828191 ip-10-0-134-209 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:56:59.828852 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.828258    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 17:56:59.831027 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831010    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:56:59.831027 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831027    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:56:59.831092 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831030    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:56:59.831092 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831033    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:56:59.831092 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831036    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:56:59.831092 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831039    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:56:59.831092 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831041    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:56:59.831092 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831044    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:56:59.831092 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831047    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:56:59.831092 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831049    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:56:59.831092 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831053    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:56:59.831092 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831055    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:56:59.831092 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831058    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:56:59.831092 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831061    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:56:59.831092 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831064    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:56:59.831092 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831067    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:56:59.831092 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831069    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:56:59.831092 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831071    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:56:59.831092 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831074    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:56:59.831092 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831078    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:56:59.831092 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831082    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:56:59.831092 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831084    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:56:59.831595 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831087    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:56:59.831595 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831089    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:56:59.831595 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831092    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:56:59.831595 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831094    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:56:59.831595 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831097    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:56:59.831595 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831100    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:56:59.831595 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831103    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:56:59.831595 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831105    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:56:59.831595 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831109    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:56:59.831595 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831113    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:56:59.831595 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831116    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:56:59.831595 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831118    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:56:59.831595 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831121    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:56:59.831595 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831123    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:56:59.831595 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831126    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:56:59.831595 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831128    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:56:59.831595 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831131    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:56:59.831595 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831133    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:56:59.831595 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831136    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:56:59.832056 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831138    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:56:59.832056 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831140    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:56:59.832056 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831142    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:56:59.832056 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831162    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:56:59.832056 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831167    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:56:59.832056 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831171    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:56:59.832056 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831174    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:56:59.832056 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831178    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:56:59.832056 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831182    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:56:59.832056 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831184    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:56:59.832056 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831187    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:56:59.832056 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831189    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:56:59.832056 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831191    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:56:59.832056 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831195    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:56:59.832056 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831199    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:56:59.832056 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831202    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:56:59.832056 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831204    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:56:59.832056 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831207    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:56:59.832056 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831209    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:56:59.832528 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831212    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:56:59.832528 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831214    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:56:59.832528 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831217    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:56:59.832528 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831219    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:56:59.832528 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831222    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:56:59.832528 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831224    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:56:59.832528 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831227    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:56:59.832528 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831229    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:56:59.832528 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831232    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:56:59.832528 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831234    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:56:59.832528 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831237    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:56:59.832528 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831241    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:56:59.832528 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831244    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:56:59.832528 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831246    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:56:59.832528 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831249    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:56:59.832528 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831251    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:56:59.832528 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831253    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:56:59.832528 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831256    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:56:59.832528 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831258    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:56:59.832528 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831260    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:56:59.833015 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831264    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:56:59.833015 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831267    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:56:59.833015 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831270    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:56:59.833015 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831273    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:56:59.833015 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831276    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:56:59.833015 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831279    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:56:59.833015 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831282    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:56:59.833015 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831286    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:56:59.833015 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831289    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:56:59.833015 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831291    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:56:59.833015 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831294    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:56:59.833015 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831297    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:56:59.833015 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831299    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:56:59.833015 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831303    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:56:59.833015 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831305    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:56:59.833015 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831308    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:56:59.833015 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831310    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:56:59.833015 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831313    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:56:59.833015 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831315    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:56:59.833015 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.831993    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:56:59.833543 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832000    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:56:59.833543 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832004    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:56:59.833543 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832007    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:56:59.833543 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832010    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:56:59.833543 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832014    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:56:59.833543 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832016    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:56:59.833543 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832019    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:56:59.833543 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832022    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:56:59.833543 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832025    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:56:59.833543 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832027    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:56:59.833543 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832030    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:56:59.833543 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832032    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:56:59.833543 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832035    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:56:59.833543 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832037    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:56:59.833543 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832039    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:56:59.833543 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832042    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:56:59.833543 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832045    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:56:59.833543 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832049    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:56:59.833543 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832053    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:56:59.833543 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832056    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:56:59.834052 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832059    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:56:59.834052 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832062    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:56:59.834052 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832065    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:56:59.834052 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832069    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:56:59.834052 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832072    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:56:59.834052 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832075    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:56:59.834052 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832078    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:56:59.834052 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832080    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:56:59.834052 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832083    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:56:59.834052 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832085    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:56:59.834052 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832087    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:56:59.834052 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832090    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:56:59.834052 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832092    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:56:59.834052 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832096    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:56:59.834052 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832098    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:56:59.834052 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832101    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:56:59.834052 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832103    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:56:59.834052 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832106    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:56:59.834052 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832108    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:56:59.834052 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832110    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:56:59.834540 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832113    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:56:59.834540 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832115    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:56:59.834540 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832117    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:56:59.834540 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832120    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:56:59.834540 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832122    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:56:59.834540 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832125    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:56:59.834540 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832127    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:56:59.834540 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832129    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:56:59.834540 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832132    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:56:59.834540 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832135    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:56:59.834540 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832137    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:56:59.834540 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832140    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:56:59.834540 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832142    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:56:59.834540 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832145    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:56:59.834540 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832162    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:56:59.834540 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832165    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:56:59.834540 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832168    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:56:59.834540 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832170    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:56:59.834540 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832173    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:56:59.834540 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832176    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:56:59.835019 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832179    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:56:59.835019 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832181    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:56:59.835019 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832183    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:56:59.835019 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832186    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:56:59.835019 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832190    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:56:59.835019 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832193    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:56:59.835019 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832197    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:56:59.835019 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832200    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:56:59.835019 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832202    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:56:59.835019 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832205    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:56:59.835019 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832208    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:56:59.835019 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832211    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:56:59.835019 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832213    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:56:59.835019 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832215    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:56:59.835019 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832218    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:56:59.835019 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832220    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:56:59.835019 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832222    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:56:59.835019 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832225    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:56:59.835019 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832227    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:56:59.835484 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832229    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:56:59.835484 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832231    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:56:59.835484 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832235    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:56:59.835484 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832237    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:56:59.835484 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832240    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:56:59.835484 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832242    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:56:59.835484 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832245    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:56:59.835484 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832247    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:56:59.835484 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832250    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:56:59.835484 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832252    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:56:59.835484 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832254    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:56:59.835484 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832257    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:56:59.835484 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832259    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:56:59.835484 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832262    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:56:59.835484 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832264    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:56:59.835484 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832266    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:56:59.835484 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832269    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:56:59.835484 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832271    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:56:59.835484 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832273    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:56:59.835484 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832604    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:56:59.835957 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832609    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:56:59.835957 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832612    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:56:59.835957 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832615    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:56:59.835957 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832618    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:56:59.835957 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832621    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:56:59.835957 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832623    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:56:59.835957 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832626    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:56:59.835957 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832628    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:56:59.835957 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832631    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:56:59.835957 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832635    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:56:59.835957 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832642    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:56:59.835957 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832645    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:56:59.835957 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832648    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:56:59.835957 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832650    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:56:59.835957 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832653    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:56:59.835957 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832655    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:56:59.835957 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832658    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:56:59.835957 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832660    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:56:59.836407 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832663    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:56:59.836407 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832665    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:56:59.836407 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832668    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:56:59.836407 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832670    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:56:59.836407 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832673    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:56:59.836407 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832675    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:56:59.836407 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832678    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:56:59.836407 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832680    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:56:59.836407 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832682    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:56:59.836407 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832687    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:56:59.836407 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832690    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:56:59.836407 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832693    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:56:59.836407 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832696    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:56:59.836407 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832699    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:56:59.836407 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832702    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:56:59.836407 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832704    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:56:59.836407 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832706    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:56:59.836407 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832709    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:56:59.836407 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832711    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:56:59.836407 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832714    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:56:59.836875 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832716    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:56:59.836875 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832719    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:56:59.836875 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832721    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:56:59.836875 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832724    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:56:59.836875 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832727    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:56:59.836875 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832729    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:56:59.836875 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832732    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:56:59.836875 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832734    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:56:59.836875 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832737    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:56:59.836875 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832739    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:56:59.836875 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832741    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:56:59.836875 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832744    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:56:59.836875 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832746    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:56:59.836875 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832749    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:56:59.836875 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832751    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:56:59.836875 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832754    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:56:59.836875 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832756    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:56:59.836875 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832758    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:56:59.836875 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832761    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:56:59.836875 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832763    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:56:59.837364 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832765    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:56:59.837364 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832768    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:56:59.837364 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832771    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:56:59.837364 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832773    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:56:59.837364 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832776    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:56:59.837364 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832778    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:56:59.837364 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832781    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:56:59.837364 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832783    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:56:59.837364 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832785    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:56:59.837364 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832788    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:56:59.837364 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832790    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:56:59.837364 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832793    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:56:59.837364 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832795    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:56:59.837364 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832797    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:56:59.837364 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832800    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:56:59.837364 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832802    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:56:59.837364 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832807    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:56:59.837364 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832809    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:56:59.837364 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832811    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:56:59.837364 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832814    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:56:59.837828 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832816    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:56:59.837828 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832818    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:56:59.837828 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832821    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:56:59.837828 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832823    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:56:59.837828 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832825    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:56:59.837828 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832828    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:56:59.837828 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832830    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:56:59.837828 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832833    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:56:59.837828 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832835    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:56:59.837828 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832838    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:56:59.837828 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832840    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:56:59.837828 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832842    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:56:59.837828 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832845    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:56:59.837828 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832847    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:56:59.837828 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832849    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:56:59.837828 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832852    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:56:59.837828 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832854    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:56:59.837828 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832858    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:56:59.837828 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832861    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:56:59.837828 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.832863    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:56:59.838329 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.832937    2581 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 17:56:59.838329 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.832970    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 17:56:59.838329 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.832977    2581 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 17:56:59.838329 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.832981    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 17:56:59.838329 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.832985    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 17:56:59.838329 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.832988    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 17:56:59.838329 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.832992    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 17:56:59.838329 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.832995    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 17:56:59.838329 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.832999    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 17:56:59.838329 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833002    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 17:56:59.838329 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833006    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 17:56:59.838329 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833009    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 17:56:59.838329 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833012    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 17:56:59.838329 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833014    2581 flags.go:64] FLAG: --cgroup-root=""
Sep 17 17:56:59.838329 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833017    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 17:56:59.838329 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833020    2581 flags.go:64] FLAG: --client-ca-file=""
Sep 17 17:56:59.838329 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833023    2581 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 17:56:59.838329 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833026    2581 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 17:56:59.838329 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833030    2581 flags.go:64] FLAG: --cluster-domain=""
Sep 17 17:56:59.838329 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833032    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 17:56:59.838329 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833035    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 17:56:59.838329 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833038    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 17:56:59.838329 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833041    2581 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 17:56:59.838329 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833045    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 17:56:59.838886 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833047    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 17:56:59.838886 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833050    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 17:56:59.838886 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833053    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 17:56:59.838886 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833056    2581 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 17:56:59.838886 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833058    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 17:56:59.838886 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833062    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 17:56:59.838886 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833065    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 17:56:59.838886 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833068    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 17:56:59.838886 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833072    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 17:56:59.838886 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833075    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 17:56:59.838886 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833077    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 17:56:59.838886 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833080    2581 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 17:56:59.838886 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833083    2581 flags.go:64] FLAG: --enable-server="true"
Sep 17 17:56:59.838886 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833086    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 17:56:59.838886 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833089    2581 flags.go:64] FLAG: --event-burst="100"
Sep 17 17:56:59.838886 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833092    2581 flags.go:64] FLAG: --event-qps="50"
Sep 17 17:56:59.838886 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833096    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 17:56:59.838886 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833099    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 17:56:59.838886 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833101    2581 flags.go:64] FLAG: --eviction-hard=""
Sep 17 17:56:59.838886 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833105    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 17:56:59.838886 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833108    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 17:56:59.838886 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833111    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 17:56:59.838886 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833113    2581 flags.go:64] FLAG: --eviction-soft=""
Sep 17 17:56:59.838886 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833116    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 17:56:59.838886 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833119    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 17:56:59.839542 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833122    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 17:56:59.839542 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833125    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 17:56:59.839542 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833127    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 17:56:59.839542 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833130    2581 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 17:56:59.839542 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833133    2581 flags.go:64] FLAG: --feature-gates=""
Sep 17 17:56:59.839542 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833136    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 17:56:59.839542 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833139    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 17:56:59.839542 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833142    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 17:56:59.839542 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833145    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 17:56:59.839542 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833168    2581 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 17:56:59.839542 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833173    2581 flags.go:64] FLAG: --help="false"
Sep 17 17:56:59.839542 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833177    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-134-209.ec2.internal"
Sep 17 17:56:59.839542 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833182    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 17:56:59.839542 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833185    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 17:56:59.839542 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833187    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 17:56:59.839542 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833190    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 17:56:59.839542 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833194    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 17:56:59.839542 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833196    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 17:56:59.839542 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833199    2581 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 17:56:59.839542 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833201    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 17:56:59.839542 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833204    2581 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 17:56:59.839542 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833208    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 17:56:59.839542 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833210    2581 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 17:56:59.839542 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833214    2581 flags.go:64] FLAG: --kube-reserved=""
Sep 17 17:56:59.840117 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833217    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 17:56:59.840117 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833219    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 17:56:59.840117 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833222    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 17:56:59.840117 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833225    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 17:56:59.840117 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833227    2581 flags.go:64] FLAG: --lock-file=""
Sep 17 17:56:59.840117 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833230    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 17:56:59.840117 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833233    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 17:56:59.840117 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833235    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 17:56:59.840117 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833240    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 17:56:59.840117 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833243    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 17:56:59.840117 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833246    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 17:56:59.840117 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833248    2581 flags.go:64] FLAG: --logging-format="text"
Sep 17 17:56:59.840117 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833251    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 17:56:59.840117 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833254    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 17:56:59.840117 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833256    2581 flags.go:64] FLAG: --manifest-url=""
Sep 17 17:56:59.840117 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833259    2581 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 17:56:59.840117 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833262    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 17:56:59.840117 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833265    2581 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 17:56:59.840117 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833269    2581 flags.go:64] FLAG: --max-pods="110"
Sep 17 17:56:59.840117 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833280    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 17:56:59.840117 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833284    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 17:56:59.840117 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833286    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 17:56:59.840117 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833289    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 17:56:59.840117 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833292    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 17:56:59.840117 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833295    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 17:56:59.840715 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833297    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 17:56:59.840715 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833304    2581 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 17:56:59.840715 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833307    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 17:56:59.840715 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833310    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 17:56:59.840715 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833313    2581 flags.go:64] FLAG: --pod-cidr=""
Sep 17 17:56:59.840715 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833316    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 17:56:59.840715 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833321    2581 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 17:56:59.840715 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833327    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 17:56:59.840715 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833330    2581 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 17:56:59.840715 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833332    2581 flags.go:64] FLAG: --port="10250"
Sep 17 17:56:59.840715 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833335    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 17:56:59.840715 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833338    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0681e376278cdb4f1"
Sep 17 17:56:59.840715 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833341    2581 flags.go:64] FLAG: --qos-reserved=""
Sep 17 17:56:59.840715 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833343    2581 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 17:56:59.840715 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833346    2581 flags.go:64] FLAG: --register-node="true"
Sep 17 17:56:59.840715 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833349    2581 flags.go:64] FLAG: --register-with-taints=""
Sep 17 17:56:59.840715 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833352    2581 flags.go:64] FLAG: --registry-burst="10"
Sep 17 17:56:59.840715 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833355    2581 flags.go:64] FLAG: --registry-qps="5"
Sep 17 17:56:59.840715 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833357    2581 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 17:56:59.840715 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833360    2581 flags.go:64] FLAG: --reserved-memory=""
Sep 17 17:56:59.840715 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833363    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 17:56:59.840715 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833366    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 17:56:59.840715 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833369    2581 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 17:56:59.840715 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833371    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 17:56:59.840715 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833374    2581 flags.go:64] FLAG: --runonce="false"
Sep 17 17:56:59.841325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833377    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 17:56:59.841325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833380    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 17:56:59.841325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833384    2581 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 17:56:59.841325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833387    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 17:56:59.841325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833389    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 17:56:59.841325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833392    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 17:56:59.841325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833395    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 17:56:59.841325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833398    2581 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 17:56:59.841325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833400    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 17:56:59.841325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833403    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 17:56:59.841325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833406    2581 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 17:56:59.841325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833408    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 17:56:59.841325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833411    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 17:56:59.841325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833414    2581 flags.go:64] FLAG: --system-cgroups=""
Sep 17 17:56:59.841325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833418    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 17:56:59.841325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833422    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 17:56:59.841325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833425    2581 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 17:56:59.841325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833428    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 17:56:59.841325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833431    2581 flags.go:64] FLAG: --tls-min-version=""
Sep 17 17:56:59.841325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833434    2581 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 17:56:59.841325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833436    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 17:56:59.841325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833439    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 17:56:59.841325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833442    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 17:56:59.841325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833445    2581 flags.go:64] FLAG: --v="2"
Sep 17 17:56:59.841325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833448    2581 flags.go:64] FLAG: --version="false"
Sep 17 17:56:59.841918 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833452    2581 flags.go:64] FLAG: --vmodule=""
Sep 17 17:56:59.841918 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833455    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 17:56:59.841918 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833458    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 17:56:59.841918 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833589    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:56:59.841918 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833593    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:56:59.841918 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833597    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:56:59.841918 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833599    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:56:59.841918 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833602    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:56:59.841918 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833605    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:56:59.841918 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833611    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:56:59.841918 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833614    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:56:59.841918 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833616    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:56:59.841918 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833619    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:56:59.841918 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833621    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:56:59.841918 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833624    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:56:59.841918 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833627    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:56:59.841918 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833629    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:56:59.841918 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833632    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:56:59.841918 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833635    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:56:59.841918 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833638    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:56:59.841918 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833640    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:56:59.842428 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833644    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:56:59.842428 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833646    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:56:59.842428 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833649    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:56:59.842428 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833652    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:56:59.842428 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833654    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:56:59.842428 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833656    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:56:59.842428 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833659    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:56:59.842428 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833661    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:56:59.842428 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833665    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:56:59.842428 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833669    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:56:59.842428 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833671    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:56:59.842428 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833675    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:56:59.842428 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833678    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:56:59.842428 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833681    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:56:59.842428 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833684    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:56:59.842428 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833687    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:56:59.842428 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833689    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:56:59.842428 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833692    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:56:59.842428 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833694    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:56:59.842927 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833697    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:56:59.842927 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833703    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:56:59.842927 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833706    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:56:59.842927 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833708    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:56:59.842927 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833711    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:56:59.842927 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833714    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:56:59.842927 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833716    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:56:59.842927 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833719    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:56:59.842927 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833721    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:56:59.842927 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833724    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:56:59.842927 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833726    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:56:59.842927 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833729    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:56:59.842927 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833731    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:56:59.842927 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833735    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:56:59.842927 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833738    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:56:59.842927 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833740    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:56:59.842927 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833742    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:56:59.842927 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833745    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:56:59.842927 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833747    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:56:59.842927 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833750    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:56:59.843404 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833752    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:56:59.843404 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833755    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:56:59.843404 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833757    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:56:59.843404 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833760    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:56:59.843404 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833762    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:56:59.843404 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833765    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:56:59.843404 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833767    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:56:59.843404 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833770    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:56:59.843404 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833772    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:56:59.843404 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833774    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:56:59.843404 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833777    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:56:59.843404 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833779    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:56:59.843404 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833782    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:56:59.843404 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833785    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:56:59.843404 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833788    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:56:59.843404 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833791    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:56:59.843404 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833793    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:56:59.843404 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833795    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:56:59.843404 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833798    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:56:59.843844 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833800    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:56:59.843844 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833803    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:56:59.843844 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833805    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:56:59.843844 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833808    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:56:59.843844 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833810    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:56:59.843844 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833812    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:56:59.843844 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833816    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:56:59.843844 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833818    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:56:59.843844 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833821    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:56:59.843844 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833823    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:56:59.843844 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833826    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:56:59.843844 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833828    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:56:59.843844 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833831    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:56:59.843844 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833833    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:56:59.843844 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833835    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:56:59.843844 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833838    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:56:59.843844 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833840    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:56:59.843844 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833843    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:56:59.843844 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833845    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:56:59.843844 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833848    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:56:59.844322 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833850    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:56:59.844322 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833852    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:56:59.844322 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.833855    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:56:59.844322 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.833859    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:56:59.844322 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.840444    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 17:56:59.844322 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.840458    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 17:56:59.844322 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.840470    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 17:56:59.844322 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.840476    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 17:56:59.844322 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840530    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:56:59.844322 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840535    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:56:59.844322 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840538    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:56:59.844322 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840541    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:56:59.844322 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840544    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:56:59.844322 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840547    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:56:59.844322 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840550    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:56:59.844686 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840553    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:56:59.844686 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840556    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:56:59.844686 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840559    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:56:59.844686 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840561    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:56:59.844686 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840564    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:56:59.844686 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840571    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:56:59.844686 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840574    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:56:59.844686 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840576    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:56:59.844686 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840579    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:56:59.844686 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840582    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:56:59.844686 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840584    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:56:59.844686 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840587    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:56:59.844686 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840589    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:56:59.844686 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840592    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:56:59.844686 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840595    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:56:59.844686 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840597    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:56:59.844686 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840600    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:56:59.844686 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840602    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:56:59.844686 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840604    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:56:59.844686 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840607    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:56:59.845171 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840611    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:56:59.845171 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840614    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:56:59.845171 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840617    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:56:59.845171 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840620    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:56:59.845171 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840623    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:56:59.845171 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840626    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:56:59.845171 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840629    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:56:59.845171 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840632    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:56:59.845171 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840634    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:56:59.845171 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840637    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:56:59.845171 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840639    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:56:59.845171 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840641    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:56:59.845171 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840644    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:56:59.845171 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840647    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:56:59.845171 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840649    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:56:59.845171 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840652    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:56:59.845171 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840654    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:56:59.845171 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840656    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:56:59.845171 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840659    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:56:59.845634 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840662    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:56:59.845634 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840665    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:56:59.845634 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840669    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:56:59.845634 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840671    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:56:59.845634 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840674    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:56:59.845634 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840677    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:56:59.845634 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840679    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:56:59.845634 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840682    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:56:59.845634 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840685    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:56:59.845634 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840687    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:56:59.845634 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840690    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:56:59.845634 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840692    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:56:59.845634 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840694    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:56:59.845634 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840697    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:56:59.845634 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840699    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:56:59.845634 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840701    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:56:59.845634 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840704    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:56:59.845634 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840706    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:56:59.845634 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840709    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:56:59.845634 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840711    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:56:59.846244 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840714    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:56:59.846244 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840716    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:56:59.846244 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840720    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:56:59.846244 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840723    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:56:59.846244 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840726    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:56:59.846244 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840729    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:56:59.846244 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840732    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:56:59.846244 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840735    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:56:59.846244 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840737    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:56:59.846244 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840740    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:56:59.846244 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840742    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:56:59.846244 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840744    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:56:59.846244 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840747    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:56:59.846244 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840750    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:56:59.846244 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840752    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:56:59.846244 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840755    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:56:59.846244 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840757    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:56:59.846244 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840760    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:56:59.846244 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840762    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:56:59.846711 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840765    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:56:59.846711 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840768    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:56:59.846711 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840770    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:56:59.846711 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840773    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:56:59.846711 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840775    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:56:59.846711 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840778    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:56:59.846711 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840780    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:56:59.846711 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840783    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:56:59.846711 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840785    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:56:59.846711 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840787    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:56:59.846711 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840790    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:56:59.846711 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840792    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:56:59.846711 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840795    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:56:59.846711 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840797    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:56:59.846711 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.840802    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:56:59.847302 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840902    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:56:59.847302 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840906    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:56:59.847302 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840909    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:56:59.847302 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840913    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:56:59.847302 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840916    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:56:59.847302 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840918    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:56:59.847302 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840921    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:56:59.847302 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840924    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:56:59.847302 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840928    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:56:59.847302 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840932    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:56:59.847302 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840935    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:56:59.847302 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840938    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:56:59.847302 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840940    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:56:59.847302 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840943    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:56:59.847302 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840946    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:56:59.847302 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840948    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:56:59.847302 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840950    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:56:59.847302 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840953    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:56:59.847302 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840955    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:56:59.847784 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840958    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:56:59.847784 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840960    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:56:59.847784 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840963    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:56:59.847784 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840965    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:56:59.847784 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840968    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:56:59.847784 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840970    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:56:59.847784 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840973    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:56:59.847784 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840976    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:56:59.847784 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840978    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:56:59.847784 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840982    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:56:59.847784 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840985    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:56:59.847784 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840988    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:56:59.847784 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840992    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:56:59.847784 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840995    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:56:59.847784 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.840998    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:56:59.847784 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841000    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:56:59.847784 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841003    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:56:59.847784 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841006    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:56:59.847784 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841008    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:56:59.848251 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841011    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:56:59.848251 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841013    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:56:59.848251 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841016    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:56:59.848251 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841018    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:56:59.848251 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841020    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:56:59.848251 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841023    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:56:59.848251 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841026    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:56:59.848251 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841028    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:56:59.848251 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841031    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:56:59.848251 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841033    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:56:59.848251 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841035    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:56:59.848251 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841038    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:56:59.848251 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841040    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:56:59.848251 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841042    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:56:59.848251 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841045    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:56:59.848251 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841047    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:56:59.848251 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841049    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:56:59.848251 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841052    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:56:59.848251 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841054    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:56:59.848251 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841056    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:56:59.848719 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841058    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:56:59.848719 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841061    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:56:59.848719 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841063    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:56:59.848719 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841065    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:56:59.848719 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841068    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:56:59.848719 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841070    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:56:59.848719 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841072    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:56:59.848719 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841075    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:56:59.848719 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841077    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:56:59.848719 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841080    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:56:59.848719 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841082    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:56:59.848719 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841084    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:56:59.848719 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841087    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:56:59.848719 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841089    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:56:59.848719 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841091    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:56:59.848719 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841094    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:56:59.848719 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841096    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:56:59.848719 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841099    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:56:59.848719 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841101    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:56:59.848719 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841104    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:56:59.849266 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841106    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:56:59.849266 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841109    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:56:59.849266 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841111    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:56:59.849266 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841113    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:56:59.849266 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841115    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:56:59.849266 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841118    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:56:59.849266 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841120    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:56:59.849266 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841123    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:56:59.849266 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841125    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:56:59.849266 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841127    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:56:59.849266 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841130    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:56:59.849266 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841132    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:56:59.849266 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841134    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:56:59.849266 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841137    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:56:59.849266 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841139    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:56:59.849266 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841142    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:56:59.849266 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841144    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:56:59.849266 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841162    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:56:59.849266 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841167    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:56:59.849266 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841169    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:56:59.849793 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:56:59.841171    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:56:59.849793 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.841176    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:56:59.849793 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.841262    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 17:56:59.849793 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.843177    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 17:56:59.849793 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.843939    2581 server.go:1019] "Starting client certificate rotation"
Sep 17 17:56:59.849793 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.844424    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:56:59.849793 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.844463    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:56:59.849793 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.847727    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:56:59.849793 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.848858    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:56:59.852549 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.852531    2581 log.go:25] "Validated CRI v1 runtime API"
Sep 17 17:56:59.853893 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.853879    2581 log.go:25] "Validated CRI v1 image API"
Sep 17 17:56:59.854539 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.854525    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 17:56:59.856505 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.856486    2581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8d8daa2e-31e7-4735-a475-d4e295d81750:/dev/nvme0n1p3 977cd0f5-386d-42ab-bb17-f752d8b93c8c:/dev/nvme0n1p4]
Sep 17 17:56:59.856567 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.856504    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 17:56:59.861250 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.861140    2581 manager.go:216] Machine: {Timestamp:2026-09-17 17:56:59.860708372 +0000 UTC m=+1.961172182 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200291 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec230888fd15273cced38223ae1fdd31 SystemUUID:ec230888-fd15-273c-ced3-8223ae1fdd31 BootID:c44f9121-676a-4d8b-827a-feb7f22198f6 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c7:4e:b4:c1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c7:4e:b4:c1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:1c:d2:ea:67:1a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 17:56:59.861250 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.861248    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 17:56:59.861358 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.861350    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 17:56:59.862707 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.862677    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 17:56:59.862819 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.862705    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-209.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 17:56:59.863781 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.863770    2581 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 17:56:59.863827 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.863783    2581 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 17:56:59.863827 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.863796    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:56:59.863827 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.863805    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:56:59.863951 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.863867    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 17:56:59.867356 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.867343    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:56:59.867461 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.867452    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 17:56:59.867493 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.867483    2581 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 17:56:59.867519 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.867492    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 17:56:59.867519 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.867505    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 17:56:59.867519 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.867513    2581 kubelet.go:398] "Adding apiserver pod source"
Sep 17 17:56:59.867618 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.867521    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 17:56:59.867875 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.867856    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:56:59.867962 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.867885    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:56:59.870654 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.870625    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 17:56:59.872875 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.872857    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 17:56:59.872953 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.872896    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 17:56:59.873966 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.873952    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 17:56:59.873966 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.873969    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 17:56:59.874067 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.873974    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 17:56:59.874067 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.873980    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 17:56:59.874067 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.873988    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 17:56:59.874067 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.873993    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 17:56:59.874067 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.874005    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 17:56:59.874067 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.874010    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 17:56:59.874067 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.874016    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 17:56:59.874067 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.874021    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 17:56:59.874067 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.874029    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 17:56:59.874067 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.874033    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 17:56:59.874067 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.874055    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 17:56:59.874067 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.874060    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 17:56:59.877779 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.877765    2581 server.go:1268] "Started kubelet"
Sep 17 17:56:59.877878 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.877824    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 17:56:59.877878 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.877834    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 17:56:59.877985 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.877893    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 17:56:59.878424 ip-10-0-134-209 systemd[1]: Started Kubernetes Kubelet.
Sep 17 17:56:59.879032 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.878991    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 17:56:59.879467 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.879403    2581 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 17:56:59.880263 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.880244    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-209.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 17:56:59.880789 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:56:59.880769    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 17:56:59.880841 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:56:59.880768    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-209.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 17:56:59.882097 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:56:59.882061    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 17:56:59.882224 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.882206    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:56:59.884372 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.884359    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 17:56:59.884431 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.884372    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 17:56:59.884431 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:56:59.884422    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-209.ec2.internal\" not found"
Sep 17 17:56:59.884848 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.884834    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 17:56:59.884924 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.884838    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 17:56:59.884924 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.884880    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 17:56:59.885014 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.884976    2581 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 17:56:59.885014 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.884984    2581 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 17:56:59.885235 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.885220    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 17:56:59.885291 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.885237    2581 factory.go:55] Registering systemd factory
Sep 17 17:56:59.885291 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.885247    2581 factory.go:223] Registration of the systemd container factory successfully
Sep 17 17:56:59.885468 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.885456    2581 factory.go:153] Registering CRI-O factory
Sep 17 17:56:59.885530 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.885471    2581 factory.go:223] Registration of the crio container factory successfully
Sep 17 17:56:59.885530 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.885485    2581 factory.go:103] Registering Raw factory
Sep 17 17:56:59.885530 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.885497    2581 manager.go:1196] Started watching for new ooms in manager
Sep 17 17:56:59.885951 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.885937    2581 manager.go:349] Starting recovery of all containers
Sep 17 17:56:59.887660 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:56:59.887638    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 17:56:59.887762 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:56:59.887720    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-209.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 17:56:59.912166 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.912125    2581 manager.go:354] Recovery completed
Sep 17 17:56:59.913441 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:56:59.912623    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-209.ec2.internal.18d62d666dc2b598  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-209.ec2.internal,UID:ip-10-0-134-209.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-209.ec2.internal,},FirstTimestamp:2026-09-17 17:56:59.877741976 +0000 UTC m=+1.978205786,LastTimestamp:2026-09-17 17:56:59.877741976 +0000 UTC m=+1.978205786,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-209.ec2.internal,}"
Sep 17 17:56:59.916130 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.916116    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:59.917795 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.917768    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:59.917857 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.917805    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:59.917857 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.917815    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-209.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:59.918413 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.918395    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 17:56:59.918413 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.918412    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 17:56:59.918525 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.918429    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:56:59.918525 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.918402    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:59.919066 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.919051    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:59.919118 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.919076    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:59.919118 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.919085    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-209.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:59.920442 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.920431    2581 policy_none.go:49] "None policy: Start"
Sep 17 17:56:59.920491 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.920446    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 17:56:59.920491 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.920455    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 17:56:59.922278 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.922268    2581 policy_none.go:47] "Start"
Sep 17 17:56:59.929223 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:56:59.929065    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-209.ec2.internal.18d62d667025d491  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-209.ec2.internal,UID:ip-10-0-134-209.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-209.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-209.ec2.internal,},FirstTimestamp:2026-09-17 17:56:59.917792401 +0000 UTC m=+2.018256211,LastTimestamp:2026-09-17 17:56:59.917792401 +0000 UTC m=+2.018256211,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-209.ec2.internal,}"
Sep 17 17:56:59.936123 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:56:59.936069    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-209.ec2.internal.18d62d66702618ee  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-209.ec2.internal,UID:ip-10-0-134-209.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-209.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-209.ec2.internal,},FirstTimestamp:2026-09-17 17:56:59.917809902 +0000 UTC m=+2.018273713,LastTimestamp:2026-09-17 17:56:59.917809902 +0000 UTC m=+2.018273713,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-209.ec2.internal,}"
Sep 17 17:56:59.941417 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:56:59.941364    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-209.ec2.internal.18d62d6670263bc8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-209.ec2.internal,UID:ip-10-0-134-209.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-209.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-209.ec2.internal,},FirstTimestamp:2026-09-17 17:56:59.917818824 +0000 UTC m=+2.018282635,LastTimestamp:2026-09-17 17:56:59.917818824 +0000 UTC m=+2.018282635,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-209.ec2.internal,}"
Sep 17 17:56:59.944321 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:56:59.944271    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-209.ec2.internal.18d62d667025d491\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-209.ec2.internal.18d62d667025d491  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-209.ec2.internal,UID:ip-10-0-134-209.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-209.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-209.ec2.internal,},FirstTimestamp:2026-09-17 17:56:59.917792401 +0000 UTC m=+2.018256211,LastTimestamp:2026-09-17 17:56:59.919064335 +0000 UTC m=+2.019528148,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-209.ec2.internal,}"
Sep 17 17:56:59.954008 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:56:59.953936    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-209.ec2.internal.18d62d66702618ee\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-209.ec2.internal.18d62d66702618ee  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-209.ec2.internal,UID:ip-10-0-134-209.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-209.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-209.ec2.internal,},FirstTimestamp:2026-09-17 17:56:59.917809902 +0000 UTC m=+2.018273713,LastTimestamp:2026-09-17 17:56:59.919080274 +0000 UTC m=+2.019544084,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-209.ec2.internal,}"
Sep 17 17:56:59.961719 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:56:59.961665    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-209.ec2.internal.18d62d6670263bc8\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-209.ec2.internal.18d62d6670263bc8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-209.ec2.internal,UID:ip-10-0-134-209.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-209.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-209.ec2.internal,},FirstTimestamp:2026-09-17 17:56:59.917818824 +0000 UTC m=+2.018282635,LastTimestamp:2026-09-17 17:56:59.919088643 +0000 UTC m=+2.019552453,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-209.ec2.internal,}"
Sep 17 17:56:59.962747 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.962736    2581 manager.go:337] "Starting Device Plugin manager"
Sep 17 17:56:59.962788 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:56:59.962772    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 17:56:59.962788 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.962781    2581 server.go:87] "Starting device plugin registration server"
Sep 17 17:56:59.963005 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.962992    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 17:56:59.963068 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.963018    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 17:56:59.963127 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.963074    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 17:56:59.963194 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.963133    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 17:56:59.963194 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:56:59.963139    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 17:56:59.963634 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:56:59.963613    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 17:56:59.963698 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:56:59.963658    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-209.ec2.internal\" not found"
Sep 17 17:56:59.987648 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:56:59.987588    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-209.ec2.internal.18d62d6673a6717f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-209.ec2.internal,UID:ip-10-0-134-209.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-134-209.ec2.internal,},FirstTimestamp:2026-09-17 17:56:59.976552831 +0000 UTC m=+2.077016631,LastTimestamp:2026-09-17 17:56:59.976552831 +0000 UTC m=+2.077016631,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-209.ec2.internal,}"
Sep 17 17:57:00.009418 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.009399    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c2zkr"
Sep 17 17:57:00.021397 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.021381    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c2zkr"
Sep 17 17:57:00.041740 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.041717    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 17:57:00.043013 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.042999    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 17:57:00.043083 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.043044    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 17:57:00.043083 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.043064    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 17:57:00.043189 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:00.043106    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 17:57:00.055358 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.055342    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:57:00.063779 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.063768    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:57:00.064561 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.064541    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:57:00.064639 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.064570    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:57:00.064639 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.064579    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-209.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:57:00.064639 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.064601    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-209.ec2.internal"
Sep 17 17:57:00.105937 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.105893    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-209.ec2.internal"
Sep 17 17:57:00.105937 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:00.105912    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-209.ec2.internal\": node \"ip-10-0-134-209.ec2.internal\" not found"
Sep 17 17:57:00.144334 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:00.144314    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-209.ec2.internal\" not found"
Sep 17 17:57:00.144436 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.144386    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-209.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-209.ec2.internal"]
Sep 17 17:57:00.144493 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.144467    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:57:00.145301 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.145287    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:57:00.145379 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.145319    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:57:00.145379 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.145336    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-209.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:57:00.147558 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.147543    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:57:00.147695 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.147681    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-209.ec2.internal"
Sep 17 17:57:00.147736 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.147714    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:57:00.148270 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.148257    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:57:00.148270 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.148266    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:57:00.148388 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.148285    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:57:00.148388 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.148290    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:57:00.148388 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.148299    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-209.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:57:00.148516 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.148300    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-209.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:57:00.150493 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.150479    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-209.ec2.internal"
Sep 17 17:57:00.150564 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.150502    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:57:00.151117 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.151098    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:57:00.151222 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.151125    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:57:00.151222 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.151165    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-209.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:57:00.179668 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:00.179649    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-209.ec2.internal\" not found" node="ip-10-0-134-209.ec2.internal"
Sep 17 17:57:00.183848 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:00.183834    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-209.ec2.internal\" not found" node="ip-10-0-134-209.ec2.internal"
Sep 17 17:57:00.185881 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.185866    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c819eb604d67aa7e331f4a25bba966ae-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-209.ec2.internal\" (UID: \"c819eb604d67aa7e331f4a25bba966ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-209.ec2.internal"
Sep 17 17:57:00.185938 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.185888    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c819eb604d67aa7e331f4a25bba966ae-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-209.ec2.internal\" (UID: \"c819eb604d67aa7e331f4a25bba966ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-209.ec2.internal"
Sep 17 17:57:00.185938 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.185903    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/284822b4778f9378098f5e86308878b3-config\") pod \"kube-apiserver-proxy-ip-10-0-134-209.ec2.internal\" (UID: \"284822b4778f9378098f5e86308878b3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-209.ec2.internal"
Sep 17 17:57:00.244946 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:00.244926    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-209.ec2.internal\" not found"
Sep 17 17:57:00.286129 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.286110    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c819eb604d67aa7e331f4a25bba966ae-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-209.ec2.internal\" (UID: \"c819eb604d67aa7e331f4a25bba966ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-209.ec2.internal"
Sep 17 17:57:00.286219 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.286132    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c819eb604d67aa7e331f4a25bba966ae-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-209.ec2.internal\" (UID: \"c819eb604d67aa7e331f4a25bba966ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-209.ec2.internal"
Sep 17 17:57:00.286219 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.286162    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/284822b4778f9378098f5e86308878b3-config\") pod \"kube-apiserver-proxy-ip-10-0-134-209.ec2.internal\" (UID: \"284822b4778f9378098f5e86308878b3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-209.ec2.internal"
Sep 17 17:57:00.286219 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.286181    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c819eb604d67aa7e331f4a25bba966ae-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-209.ec2.internal\" (UID: \"c819eb604d67aa7e331f4a25bba966ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-209.ec2.internal"
Sep 17 17:57:00.286219 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.286191    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c819eb604d67aa7e331f4a25bba966ae-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-209.ec2.internal\" (UID: \"c819eb604d67aa7e331f4a25bba966ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-209.ec2.internal"
Sep 17 17:57:00.286219 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.286191    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/284822b4778f9378098f5e86308878b3-config\") pod \"kube-apiserver-proxy-ip-10-0-134-209.ec2.internal\" (UID: \"284822b4778f9378098f5e86308878b3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-209.ec2.internal"
Sep 17 17:57:00.345727 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:00.345695    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-209.ec2.internal\" not found"
Sep 17 17:57:00.446612 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:00.446572    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-209.ec2.internal\" not found"
Sep 17 17:57:00.482743 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.482717    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-209.ec2.internal"
Sep 17 17:57:00.487871 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.487842    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-209.ec2.internal"
Sep 17 17:57:00.546923 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:00.546896    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-209.ec2.internal\" not found"
Sep 17 17:57:00.647384 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:00.647357    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-209.ec2.internal\" not found"
Sep 17 17:57:00.747862 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:00.747799    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-209.ec2.internal\" not found"
Sep 17 17:57:00.844306 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.844282    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 17:57:00.844701 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.844418    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:57:00.848432 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:00.848410    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-209.ec2.internal\" not found"
Sep 17 17:57:00.862523 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.862498    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:57:00.909521 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.909501    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:57:00.985430 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:00.985405    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-209.ec2.internal"
Sep 17 17:57:01.003717 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.003669    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:57:01.004522 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.004505    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-209.ec2.internal"
Sep 17 17:57:01.016775 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.016752    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:57:01.026910 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.026876    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:52:00 +0000 UTC" deadline="2028-06-22 19:11:58.977293896 +0000 UTC"
Sep 17 17:57:01.026910 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.026907    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15457h14m57.950391228s"
Sep 17 17:57:01.109576 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:57:01.109543    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod284822b4778f9378098f5e86308878b3.slice/crio-03427fa498b8b7fb4b131db54165c84b0d895ec2ca301d6e1896709b7421ac6d WatchSource:0}: Error finding container 03427fa498b8b7fb4b131db54165c84b0d895ec2ca301d6e1896709b7421ac6d: Status 404 returned error can't find the container with id 03427fa498b8b7fb4b131db54165c84b0d895ec2ca301d6e1896709b7421ac6d
Sep 17 17:57:01.109921 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:57:01.109896    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc819eb604d67aa7e331f4a25bba966ae.slice/crio-3864486d831b8b1a3635890ad854c5094b6f23e00e467a37587155e9b9a64dec WatchSource:0}: Error finding container 3864486d831b8b1a3635890ad854c5094b6f23e00e467a37587155e9b9a64dec: Status 404 returned error can't find the container with id 3864486d831b8b1a3635890ad854c5094b6f23e00e467a37587155e9b9a64dec
Sep 17 17:57:01.114885 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.114871    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:57:01.433544 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.433474    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:57:01.869167 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.869066    2581 apiserver.go:52] "Watching apiserver"
Sep 17 17:57:01.874586 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.874563    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 17:57:01.874971 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.874949    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-2wq8v","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-209.ec2.internal","openshift-multus/network-metrics-daemon-dsz4t","openshift-network-diagnostics/network-check-target-98c4r","kube-system/global-pull-secret-syncer-shvtg","openshift-image-registry/node-ca-k7mgr","openshift-multus/multus-additional-cni-plugins-8qdrq","openshift-multus/multus-xhbcn","openshift-network-operator/iptables-alerter-4qkk8","openshift-ovn-kubernetes/ovnkube-node-qm8hx","kube-system/konnectivity-agent-sf7g4","kube-system/kube-apiserver-proxy-ip-10-0-134-209.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7czlw","openshift-cluster-node-tuning-operator/tuned-mvn46"]
Sep 17 17:57:01.877935 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.877449    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8qdrq"
Sep 17 17:57:01.879613 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.879594    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7czlw"
Sep 17 17:57:01.881451 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.881432    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fbc9q\""
Sep 17 17:57:01.883677 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.883660    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 17:57:01.886391 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.886369    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:57:01.886479 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:01.886444    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsz4t" podUID="11b6807f-cb26-4117-b06d-eb974e5640ae"
Sep 17 17:57:01.886599 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.886586    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 17:57:01.888011 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.887993    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 17:57:01.888240 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.888221    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 17:57:01.888469 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.888448    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 17:57:01.889755 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.888633    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 17:57:01.889755 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.888869    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p88j8\""
Sep 17 17:57:01.889755 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.889058    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 17:57:01.889755 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.889256    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 17:57:01.889755 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.889447    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 17:57:01.889755 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.889675    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:57:01.889755 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:01.889721    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98c4r" podUID="c53ddf27-1765-4785-b122-1645ccc8d094"
Sep 17 17:57:01.890634 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.890614    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:01.890722 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:01.890674    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shvtg" podUID="41ab02fa-694e-4b41-8969-0e5181664b3a"
Sep 17 17:57:01.894279 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.893781    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k7mgr"
Sep 17 17:57:01.894279 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.893807    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/38076ecb-73ae-4c1c-8359-b9e2b7435a7b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8qdrq\" (UID: \"38076ecb-73ae-4c1c-8359-b9e2b7435a7b\") " pod="openshift-multus/multus-additional-cni-plugins-8qdrq"
Sep 17 17:57:01.894279 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.893835    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plxzl\" (UniqueName: \"kubernetes.io/projected/38076ecb-73ae-4c1c-8359-b9e2b7435a7b-kube-api-access-plxzl\") pod \"multus-additional-cni-plugins-8qdrq\" (UID: \"38076ecb-73ae-4c1c-8359-b9e2b7435a7b\") " pod="openshift-multus/multus-additional-cni-plugins-8qdrq"
Sep 17 17:57:01.894279 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.893862    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ddb24cd0-f80d-426b-9910-71b2b1a4d578-device-dir\") pod \"aws-ebs-csi-driver-node-7czlw\" (UID: \"ddb24cd0-f80d-426b-9910-71b2b1a4d578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7czlw"
Sep 17 17:57:01.894279 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.893878    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11b6807f-cb26-4117-b06d-eb974e5640ae-metrics-certs\") pod \"network-metrics-daemon-dsz4t\" (UID: \"11b6807f-cb26-4117-b06d-eb974e5640ae\") " pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:57:01.894279 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.893894    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/38076ecb-73ae-4c1c-8359-b9e2b7435a7b-cnibin\") pod \"multus-additional-cni-plugins-8qdrq\" (UID: \"38076ecb-73ae-4c1c-8359-b9e2b7435a7b\") " pod="openshift-multus/multus-additional-cni-plugins-8qdrq"
Sep 17 17:57:01.894279 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.893918    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ddb24cd0-f80d-426b-9910-71b2b1a4d578-socket-dir\") pod \"aws-ebs-csi-driver-node-7czlw\" (UID: \"ddb24cd0-f80d-426b-9910-71b2b1a4d578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7czlw"
Sep 17 17:57:01.894279 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.893940    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2hn2\" (UniqueName: \"kubernetes.io/projected/c53ddf27-1765-4785-b122-1645ccc8d094-kube-api-access-p2hn2\") pod \"network-check-target-98c4r\" (UID: \"c53ddf27-1765-4785-b122-1645ccc8d094\") " pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:57:01.894279 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.893965    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ddb24cd0-f80d-426b-9910-71b2b1a4d578-registration-dir\") pod \"aws-ebs-csi-driver-node-7czlw\" (UID: \"ddb24cd0-f80d-426b-9910-71b2b1a4d578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7czlw"
Sep 17 17:57:01.894279 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.893988    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ddb24cd0-f80d-426b-9910-71b2b1a4d578-sys-fs\") pod \"aws-ebs-csi-driver-node-7czlw\" (UID: \"ddb24cd0-f80d-426b-9910-71b2b1a4d578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7czlw"
Sep 17 17:57:01.894279 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.894009    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5kvg\" (UniqueName: \"kubernetes.io/projected/ddb24cd0-f80d-426b-9910-71b2b1a4d578-kube-api-access-p5kvg\") pod \"aws-ebs-csi-driver-node-7czlw\" (UID: \"ddb24cd0-f80d-426b-9910-71b2b1a4d578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7czlw"
Sep 17 17:57:01.894279 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.894032    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/38076ecb-73ae-4c1c-8359-b9e2b7435a7b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8qdrq\" (UID: \"38076ecb-73ae-4c1c-8359-b9e2b7435a7b\") " pod="openshift-multus/multus-additional-cni-plugins-8qdrq"
Sep 17 17:57:01.894279 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.894055    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/38076ecb-73ae-4c1c-8359-b9e2b7435a7b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8qdrq\" (UID: \"38076ecb-73ae-4c1c-8359-b9e2b7435a7b\") " pod="openshift-multus/multus-additional-cni-plugins-8qdrq"
Sep 17 17:57:01.894279 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.894078    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ddb24cd0-f80d-426b-9910-71b2b1a4d578-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7czlw\" (UID: \"ddb24cd0-f80d-426b-9910-71b2b1a4d578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7czlw"
Sep 17 17:57:01.894279 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.894101    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ddb24cd0-f80d-426b-9910-71b2b1a4d578-etc-selinux\") pod \"aws-ebs-csi-driver-node-7czlw\" (UID: \"ddb24cd0-f80d-426b-9910-71b2b1a4d578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7czlw"
Sep 17 17:57:01.894279 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.894123    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgfw5\" (UniqueName: \"kubernetes.io/projected/11b6807f-cb26-4117-b06d-eb974e5640ae-kube-api-access-dgfw5\") pod \"network-metrics-daemon-dsz4t\" (UID: \"11b6807f-cb26-4117-b06d-eb974e5640ae\") " pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:57:01.895070 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.894145    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/38076ecb-73ae-4c1c-8359-b9e2b7435a7b-system-cni-dir\") pod \"multus-additional-cni-plugins-8qdrq\" (UID: \"38076ecb-73ae-4c1c-8359-b9e2b7435a7b\") " pod="openshift-multus/multus-additional-cni-plugins-8qdrq"
Sep 17 17:57:01.895070 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.894202    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/38076ecb-73ae-4c1c-8359-b9e2b7435a7b-cni-binary-copy\") pod \"multus-additional-cni-plugins-8qdrq\" (UID: \"38076ecb-73ae-4c1c-8359-b9e2b7435a7b\") " pod="openshift-multus/multus-additional-cni-plugins-8qdrq"
Sep 17 17:57:01.896174 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.896083    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4qkk8"
Sep 17 17:57:01.898380 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.898362    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2wq8v"
Sep 17 17:57:01.900173 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.900006    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:57:01.900173 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.900043    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 17:57:01.900321 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.900295    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 17:57:01.900574 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.900555    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 17:57:01.900633 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.900594    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8cf8f\""
Sep 17 17:57:01.900847 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.900791    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-642zt\""
Sep 17 17:57:01.900915 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.900869    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9ddlj\""
Sep 17 17:57:01.900956 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.900884    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 17:57:01.901745 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.901533    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:01.901745 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.901571    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 17:57:01.902178 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.902139    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 17:57:01.905408 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.903661    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 17:57:01.905408 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.904846    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sf7g4"
Sep 17 17:57:01.906845 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.906824    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 17:57:01.907796 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.907490    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:01.907796 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.907514    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 17:57:01.909728 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.908810    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 17:57:01.909857 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.909756    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:01.910540 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.910501    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 17:57:01.910540 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.910515    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5r25n\""
Sep 17 17:57:01.910712 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.910695    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bfgxz\""
Sep 17 17:57:01.910773 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.910752    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 17:57:01.912117 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.912102    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 17:57:01.912337 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.912307    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 17:57:01.912910 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.912879    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 17:57:01.917950 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.917930    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 17:57:01.918208 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.918191    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 17:57:01.919124 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.918371    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:57:01.919124 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.918620    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-v9dhj\""
Sep 17 17:57:01.919124 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.918814    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:57:01.921199 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.921181    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ddrtm\""
Sep 17 17:57:01.938272 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.938254    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wxgdb"
Sep 17 17:57:01.957249 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.957215    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wxgdb"
Sep 17 17:57:01.986623 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.986595    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 17:57:01.994838 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.994816    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ddb24cd0-f80d-426b-9910-71b2b1a4d578-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7czlw\" (UID: \"ddb24cd0-f80d-426b-9910-71b2b1a4d578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7czlw"
Sep 17 17:57:01.994966 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.994849    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ddb24cd0-f80d-426b-9910-71b2b1a4d578-etc-selinux\") pod \"aws-ebs-csi-driver-node-7czlw\" (UID: \"ddb24cd0-f80d-426b-9910-71b2b1a4d578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7czlw"
Sep 17 17:57:01.994966 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.994876    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/38076ecb-73ae-4c1c-8359-b9e2b7435a7b-cni-binary-copy\") pod \"multus-additional-cni-plugins-8qdrq\" (UID: \"38076ecb-73ae-4c1c-8359-b9e2b7435a7b\") " pod="openshift-multus/multus-additional-cni-plugins-8qdrq"
Sep 17 17:57:01.994966 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.994897    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ddb24cd0-f80d-426b-9910-71b2b1a4d578-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7czlw\" (UID: \"ddb24cd0-f80d-426b-9910-71b2b1a4d578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7czlw"
Sep 17 17:57:01.994966 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.994903    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6345bbfb-99aa-42ba-b5c9-d7c46ef2392f-hosts-file\") pod \"node-resolver-2wq8v\" (UID: \"6345bbfb-99aa-42ba-b5c9-d7c46ef2392f\") " pod="openshift-dns/node-resolver-2wq8v"
Sep 17 17:57:01.994966 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.994923    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-host-kubelet\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:01.994966 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.994945    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-etc-kubernetes\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:01.995216 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.994990    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-multus-conf-dir\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:01.995216 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995032    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11b6807f-cb26-4117-b06d-eb974e5640ae-metrics-certs\") pod \"network-metrics-daemon-dsz4t\" (UID: \"11b6807f-cb26-4117-b06d-eb974e5640ae\") " pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:57:01.995216 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995036    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ddb24cd0-f80d-426b-9910-71b2b1a4d578-etc-selinux\") pod \"aws-ebs-csi-driver-node-7czlw\" (UID: \"ddb24cd0-f80d-426b-9910-71b2b1a4d578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7czlw"
Sep 17 17:57:01.995216 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995075    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-host-run-ovn-kubernetes\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:01.995216 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995103    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-host-cni-bin\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:01.995216 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:01.995124    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:01.995216 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995126    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-system-cni-dir\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:01.995216 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:01.995209    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11b6807f-cb26-4117-b06d-eb974e5640ae-metrics-certs podName:11b6807f-cb26-4117-b06d-eb974e5640ae nodeName:}" failed. No retries permitted until 2026-09-17 17:57:02.495185899 +0000 UTC m=+4.595649703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11b6807f-cb26-4117-b06d-eb974e5640ae-metrics-certs") pod "network-metrics-daemon-dsz4t" (UID: "11b6807f-cb26-4117-b06d-eb974e5640ae") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:01.995536 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995227    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ddb24cd0-f80d-426b-9910-71b2b1a4d578-socket-dir\") pod \"aws-ebs-csi-driver-node-7czlw\" (UID: \"ddb24cd0-f80d-426b-9910-71b2b1a4d578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7czlw"
Sep 17 17:57:01.995536 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995254    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-run-systemd\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:01.995536 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995276    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-etc-sysconfig\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:01.995536 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-sys\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:01.995536 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995319    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ba4edf3-c791-4134-9ea3-449a5c3b437c-tmp\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:01.995536 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995328    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ddb24cd0-f80d-426b-9910-71b2b1a4d578-socket-dir\") pod \"aws-ebs-csi-driver-node-7czlw\" (UID: \"ddb24cd0-f80d-426b-9910-71b2b1a4d578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7czlw"
Sep 17 17:57:01.995536 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995343    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/41ab02fa-694e-4b41-8969-0e5181664b3a-kubelet-config\") pod \"global-pull-secret-syncer-shvtg\" (UID: \"41ab02fa-694e-4b41-8969-0e5181664b3a\") " pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:01.995536 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995377    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ddb24cd0-f80d-426b-9910-71b2b1a4d578-sys-fs\") pod \"aws-ebs-csi-driver-node-7czlw\" (UID: \"ddb24cd0-f80d-426b-9910-71b2b1a4d578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7czlw"
Sep 17 17:57:01.995536 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995409    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5kvg\" (UniqueName: \"kubernetes.io/projected/ddb24cd0-f80d-426b-9910-71b2b1a4d578-kube-api-access-p5kvg\") pod \"aws-ebs-csi-driver-node-7czlw\" (UID: \"ddb24cd0-f80d-426b-9910-71b2b1a4d578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7czlw"
Sep 17 17:57:01.995536 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995456    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ddb24cd0-f80d-426b-9910-71b2b1a4d578-sys-fs\") pod \"aws-ebs-csi-driver-node-7czlw\" (UID: \"ddb24cd0-f80d-426b-9910-71b2b1a4d578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7czlw"
Sep 17 17:57:01.995536 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995491    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/38076ecb-73ae-4c1c-8359-b9e2b7435a7b-cni-binary-copy\") pod \"multus-additional-cni-plugins-8qdrq\" (UID: \"38076ecb-73ae-4c1c-8359-b9e2b7435a7b\") " pod="openshift-multus/multus-additional-cni-plugins-8qdrq"
Sep 17 17:57:01.995536 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/38076ecb-73ae-4c1c-8359-b9e2b7435a7b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8qdrq\" (UID: \"38076ecb-73ae-4c1c-8359-b9e2b7435a7b\") " pod="openshift-multus/multus-additional-cni-plugins-8qdrq"
Sep 17 17:57:01.995974 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995601    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-ovn-node-metrics-cert\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:01.995974 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995636    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-etc-systemd\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:01.995974 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995663    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plxzl\" (UniqueName: \"kubernetes.io/projected/38076ecb-73ae-4c1c-8359-b9e2b7435a7b-kube-api-access-plxzl\") pod \"multus-additional-cni-plugins-8qdrq\" (UID: \"38076ecb-73ae-4c1c-8359-b9e2b7435a7b\") " pod="openshift-multus/multus-additional-cni-plugins-8qdrq"
Sep 17 17:57:01.995974 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995697    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-node-log\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:01.995974 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995720    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njd8d\" (UniqueName: \"kubernetes.io/projected/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-kube-api-access-njd8d\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:01.995974 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995743    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ca53e15c-51d9-40c8-9d87-80e217ce1815-konnectivity-ca\") pod \"konnectivity-agent-sf7g4\" (UID: \"ca53e15c-51d9-40c8-9d87-80e217ce1815\") " pod="kube-system/konnectivity-agent-sf7g4"
Sep 17 17:57:01.995974 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995766    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-multus-cni-dir\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:01.995974 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995805    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzq5m\" (UniqueName: \"kubernetes.io/projected/bd3ffd9f-338b-48ca-97c2-4292e8716182-kube-api-access-vzq5m\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:01.995974 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995828    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-etc-modprobe-d\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:01.995974 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995852    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b4f2a63c-9017-431f-bec9-47521af4d1bd-iptables-alerter-script\") pod \"iptables-alerter-4qkk8\" (UID: \"b4f2a63c-9017-431f-bec9-47521af4d1bd\") " pod="openshift-network-operator/iptables-alerter-4qkk8"
Sep 17 17:57:01.995974 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995879    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41ab02fa-694e-4b41-8969-0e5181664b3a-original-pull-secret\") pod \"global-pull-secret-syncer-shvtg\" (UID: \"41ab02fa-694e-4b41-8969-0e5181664b3a\") " pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:01.995974 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995902    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/383b592e-5a46-4299-9522-ac835c0adc61-serviceca\") pod \"node-ca-k7mgr\" (UID: \"383b592e-5a46-4299-9522-ac835c0adc61\") " pod="openshift-image-registry/node-ca-k7mgr"
Sep 17 17:57:01.995974 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995920    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/38076ecb-73ae-4c1c-8359-b9e2b7435a7b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8qdrq\" (UID: \"38076ecb-73ae-4c1c-8359-b9e2b7435a7b\") " pod="openshift-multus/multus-additional-cni-plugins-8qdrq"
Sep 17 17:57:01.995974 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995946    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:01.996658 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.995984    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-ovnkube-config\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:01.996658 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996005    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7ba4edf3-c791-4134-9ea3-449a5c3b437c-etc-tuned\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:01.996658 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-host-run-netns\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:01.996658 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996067    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2hn2\" (UniqueName: \"kubernetes.io/projected/c53ddf27-1765-4785-b122-1645ccc8d094-kube-api-access-p2hn2\") pod \"network-check-target-98c4r\" (UID: \"c53ddf27-1765-4785-b122-1645ccc8d094\") " pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:57:01.996658 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996107    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-host-slash\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:01.996658 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996132    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-host-run-netns\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:01.996658 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996180    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ca53e15c-51d9-40c8-9d87-80e217ce1815-agent-certs\") pod \"konnectivity-agent-sf7g4\" (UID: \"ca53e15c-51d9-40c8-9d87-80e217ce1815\") " pod="kube-system/konnectivity-agent-sf7g4"
Sep 17 17:57:01.996658 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996224    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-etc-sysctl-conf\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:01.996658 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996246    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfc5w\" (UniqueName: \"kubernetes.io/projected/7ba4edf3-c791-4134-9ea3-449a5c3b437c-kube-api-access-wfc5w\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:01.996658 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996271    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-host-var-lib-kubelet\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:01.996658 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996293    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-run-ovn\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:01.996658 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996315    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-env-overrides\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:01.996658 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996339    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-var-lib-kubelet\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:01.996658 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996378    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd3ffd9f-338b-48ca-97c2-4292e8716182-cni-binary-copy\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:01.996658 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996401    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-host-run-multus-certs\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:01.996658 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996427    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ddb24cd0-f80d-426b-9910-71b2b1a4d578-device-dir\") pod \"aws-ebs-csi-driver-node-7czlw\" (UID: \"ddb24cd0-f80d-426b-9910-71b2b1a4d578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7czlw"
Sep 17 17:57:01.996658 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996451    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-ovnkube-script-lib\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:01.997442 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996475    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-host-run-k8s-cni-cncf-io\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:01.997442 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996497    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqhgj\" (UniqueName: \"kubernetes.io/projected/b4f2a63c-9017-431f-bec9-47521af4d1bd-kube-api-access-xqhgj\") pod \"iptables-alerter-4qkk8\" (UID: \"b4f2a63c-9017-431f-bec9-47521af4d1bd\") " pod="openshift-network-operator/iptables-alerter-4qkk8"
Sep 17 17:57:01.997442 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996520    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8kfc\" (UniqueName: \"kubernetes.io/projected/6345bbfb-99aa-42ba-b5c9-d7c46ef2392f-kube-api-access-r8kfc\") pod \"node-resolver-2wq8v\" (UID: \"6345bbfb-99aa-42ba-b5c9-d7c46ef2392f\") " pod="openshift-dns/node-resolver-2wq8v"
Sep 17 17:57:01.997442 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996528    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ddb24cd0-f80d-426b-9910-71b2b1a4d578-device-dir\") pod \"aws-ebs-csi-driver-node-7czlw\" (UID: \"ddb24cd0-f80d-426b-9910-71b2b1a4d578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7czlw"
Sep 17 17:57:01.997442 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996544    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-lib-modules\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:01.997442 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996564    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-host\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:01.997442 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996588    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/38076ecb-73ae-4c1c-8359-b9e2b7435a7b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8qdrq\" (UID: \"38076ecb-73ae-4c1c-8359-b9e2b7435a7b\") " pod="openshift-multus/multus-additional-cni-plugins-8qdrq"
Sep 17 17:57:01.997442 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996610    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jsgj\" (UniqueName: \"kubernetes.io/projected/383b592e-5a46-4299-9522-ac835c0adc61-kube-api-access-6jsgj\") pod \"node-ca-k7mgr\" (UID: \"383b592e-5a46-4299-9522-ac835c0adc61\") " pod="openshift-image-registry/node-ca-k7mgr"
Sep 17 17:57:01.997442 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996635    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-systemd-units\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:01.997442 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996656    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-log-socket\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:01.997442 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996678    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-cnibin\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:01.997442 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996701    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-hostroot\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:01.997442 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996725    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgfw5\" (UniqueName: \"kubernetes.io/projected/11b6807f-cb26-4117-b06d-eb974e5640ae-kube-api-access-dgfw5\") pod \"network-metrics-daemon-dsz4t\" (UID: \"11b6807f-cb26-4117-b06d-eb974e5640ae\") " pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:57:01.997442 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996731    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/38076ecb-73ae-4c1c-8359-b9e2b7435a7b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8qdrq\" (UID: \"38076ecb-73ae-4c1c-8359-b9e2b7435a7b\") " pod="openshift-multus/multus-additional-cni-plugins-8qdrq"
Sep 17 17:57:01.997442 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996757    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/38076ecb-73ae-4c1c-8359-b9e2b7435a7b-system-cni-dir\") pod \"multus-additional-cni-plugins-8qdrq\" (UID: \"38076ecb-73ae-4c1c-8359-b9e2b7435a7b\") " pod="openshift-multus/multus-additional-cni-plugins-8qdrq"
Sep 17 17:57:01.997442 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996831    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/38076ecb-73ae-4c1c-8359-b9e2b7435a7b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8qdrq\" (UID: \"38076ecb-73ae-4c1c-8359-b9e2b7435a7b\") " pod="openshift-multus/multus-additional-cni-plugins-8qdrq"
Sep 17 17:57:01.997951 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996839    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/38076ecb-73ae-4c1c-8359-b9e2b7435a7b-system-cni-dir\") pod \"multus-additional-cni-plugins-8qdrq\" (UID: \"38076ecb-73ae-4c1c-8359-b9e2b7435a7b\") " pod="openshift-multus/multus-additional-cni-plugins-8qdrq"
Sep 17 17:57:01.997951 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6345bbfb-99aa-42ba-b5c9-d7c46ef2392f-tmp-dir\") pod \"node-resolver-2wq8v\" (UID: \"6345bbfb-99aa-42ba-b5c9-d7c46ef2392f\") " pod="openshift-dns/node-resolver-2wq8v"
Sep 17 17:57:01.997951 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996913    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-var-lib-openvswitch\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:01.997951 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996953    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd3ffd9f-338b-48ca-97c2-4292e8716182-multus-daemon-config\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:01.997951 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.996982    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/38076ecb-73ae-4c1c-8359-b9e2b7435a7b-cnibin\") pod \"multus-additional-cni-plugins-8qdrq\" (UID: \"38076ecb-73ae-4c1c-8359-b9e2b7435a7b\") " pod="openshift-multus/multus-additional-cni-plugins-8qdrq"
Sep 17 17:57:01.997951 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.997014    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b4f2a63c-9017-431f-bec9-47521af4d1bd-host-slash\") pod \"iptables-alerter-4qkk8\" (UID: \"b4f2a63c-9017-431f-bec9-47521af4d1bd\") " pod="openshift-network-operator/iptables-alerter-4qkk8"
Sep 17 17:57:01.997951 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.997030    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/41ab02fa-694e-4b41-8969-0e5181664b3a-dbus\") pod \"global-pull-secret-syncer-shvtg\" (UID: \"41ab02fa-694e-4b41-8969-0e5181664b3a\") " pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:01.997951 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.997044    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-etc-openvswitch\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:01.997951 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.997050    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/38076ecb-73ae-4c1c-8359-b9e2b7435a7b-cnibin\") pod \"multus-additional-cni-plugins-8qdrq\" (UID: \"38076ecb-73ae-4c1c-8359-b9e2b7435a7b\") " pod="openshift-multus/multus-additional-cni-plugins-8qdrq"
Sep 17 17:57:01.997951 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.997058    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-run\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:01.997951 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.997084    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-host-var-lib-cni-multus\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:01.997951 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.997143    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/383b592e-5a46-4299-9522-ac835c0adc61-host\") pod \"node-ca-k7mgr\" (UID: \"383b592e-5a46-4299-9522-ac835c0adc61\") " pod="openshift-image-registry/node-ca-k7mgr"
Sep 17 17:57:01.997951 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.997190    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-run-openvswitch\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:01.997951 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.997215    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-host-var-lib-cni-bin\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:01.997951 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.997245    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-etc-kubernetes\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:01.997951 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.997276    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ddb24cd0-f80d-426b-9910-71b2b1a4d578-registration-dir\") pod \"aws-ebs-csi-driver-node-7czlw\" (UID: \"ddb24cd0-f80d-426b-9910-71b2b1a4d578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7czlw"
Sep 17 17:57:01.997951 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.997308    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-host-cni-netd\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:01.998486 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.997331    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-etc-sysctl-d\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:01.998486 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.997354    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-multus-socket-dir-parent\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:01.998486 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.997356    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/38076ecb-73ae-4c1c-8359-b9e2b7435a7b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8qdrq\" (UID: \"38076ecb-73ae-4c1c-8359-b9e2b7435a7b\") " pod="openshift-multus/multus-additional-cni-plugins-8qdrq"
Sep 17 17:57:01.998486 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:01.997359    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ddb24cd0-f80d-426b-9910-71b2b1a4d578-registration-dir\") pod \"aws-ebs-csi-driver-node-7czlw\" (UID: \"ddb24cd0-f80d-426b-9910-71b2b1a4d578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7czlw"
Sep 17 17:57:02.010137 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.010106    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 17:57:02.012283 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:02.012260    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:57:02.012283 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:02.012286    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:57:02.012423 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:02.012302    2581 projected.go:196] Error preparing data for projected volume kube-api-access-p2hn2 for pod openshift-network-diagnostics/network-check-target-98c4r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:02.012423 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:02.012364    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c53ddf27-1765-4785-b122-1645ccc8d094-kube-api-access-p2hn2 podName:c53ddf27-1765-4785-b122-1645ccc8d094 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:02.512347576 +0000 UTC m=+4.612811379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-p2hn2" (UniqueName: "kubernetes.io/projected/c53ddf27-1765-4785-b122-1645ccc8d094-kube-api-access-p2hn2") pod "network-check-target-98c4r" (UID: "c53ddf27-1765-4785-b122-1645ccc8d094") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:02.013923 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.013897    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5kvg\" (UniqueName: \"kubernetes.io/projected/ddb24cd0-f80d-426b-9910-71b2b1a4d578-kube-api-access-p5kvg\") pod \"aws-ebs-csi-driver-node-7czlw\" (UID: \"ddb24cd0-f80d-426b-9910-71b2b1a4d578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7czlw"
Sep 17 17:57:02.014002 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.013942    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plxzl\" (UniqueName: \"kubernetes.io/projected/38076ecb-73ae-4c1c-8359-b9e2b7435a7b-kube-api-access-plxzl\") pod \"multus-additional-cni-plugins-8qdrq\" (UID: \"38076ecb-73ae-4c1c-8359-b9e2b7435a7b\") " pod="openshift-multus/multus-additional-cni-plugins-8qdrq"
Sep 17 17:57:02.014081 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.014061    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgfw5\" (UniqueName: \"kubernetes.io/projected/11b6807f-cb26-4117-b06d-eb974e5640ae-kube-api-access-dgfw5\") pod \"network-metrics-daemon-dsz4t\" (UID: \"11b6807f-cb26-4117-b06d-eb974e5640ae\") " pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:57:02.047420 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.047376    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-209.ec2.internal" event={"ID":"284822b4778f9378098f5e86308878b3","Type":"ContainerStarted","Data":"03427fa498b8b7fb4b131db54165c84b0d895ec2ca301d6e1896709b7421ac6d"}
Sep 17 17:57:02.048558 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.048528    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-209.ec2.internal" event={"ID":"c819eb604d67aa7e331f4a25bba966ae","Type":"ContainerStarted","Data":"3864486d831b8b1a3635890ad854c5094b6f23e00e467a37587155e9b9a64dec"}
Sep 17 17:57:02.098505 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.098473    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8kfc\" (UniqueName: \"kubernetes.io/projected/6345bbfb-99aa-42ba-b5c9-d7c46ef2392f-kube-api-access-r8kfc\") pod \"node-resolver-2wq8v\" (UID: \"6345bbfb-99aa-42ba-b5c9-d7c46ef2392f\") " pod="openshift-dns/node-resolver-2wq8v"
Sep 17 17:57:02.098505 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.098507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-lib-modules\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:02.098698 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.098522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-host\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:02.098698 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.098543    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jsgj\" (UniqueName: \"kubernetes.io/projected/383b592e-5a46-4299-9522-ac835c0adc61-kube-api-access-6jsgj\") pod \"node-ca-k7mgr\" (UID: \"383b592e-5a46-4299-9522-ac835c0adc61\") " pod="openshift-image-registry/node-ca-k7mgr"
Sep 17 17:57:02.098698 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.098565    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-systemd-units\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.098698 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.098610    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-systemd-units\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.098698 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.098633    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-host\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:02.098698 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.098680    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-log-socket\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.098902 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.098706    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-cnibin\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.098902 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.098721    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-hostroot\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.098902 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.098737    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6345bbfb-99aa-42ba-b5c9-d7c46ef2392f-tmp-dir\") pod \"node-resolver-2wq8v\" (UID: \"6345bbfb-99aa-42ba-b5c9-d7c46ef2392f\") " pod="openshift-dns/node-resolver-2wq8v"
Sep 17 17:57:02.098902 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.098754    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-var-lib-openvswitch\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.098902 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.098778    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd3ffd9f-338b-48ca-97c2-4292e8716182-multus-daemon-config\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.098902 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.098798    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b4f2a63c-9017-431f-bec9-47521af4d1bd-host-slash\") pod \"iptables-alerter-4qkk8\" (UID: \"b4f2a63c-9017-431f-bec9-47521af4d1bd\") " pod="openshift-network-operator/iptables-alerter-4qkk8"
Sep 17 17:57:02.098902 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.098819    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-cnibin\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.098902 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.098830    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/41ab02fa-694e-4b41-8969-0e5181664b3a-dbus\") pod \"global-pull-secret-syncer-shvtg\" (UID: \"41ab02fa-694e-4b41-8969-0e5181664b3a\") " pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:02.098902 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.098853    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-etc-openvswitch\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.098902 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.098862    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-log-socket\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.098902 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.098874    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-run\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:02.098902 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.098883    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-var-lib-openvswitch\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.098902 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.098888    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-host-var-lib-cni-multus\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.098902 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.098897    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-hostroot\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.099390 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.098917    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/383b592e-5a46-4299-9522-ac835c0adc61-host\") pod \"node-ca-k7mgr\" (UID: \"383b592e-5a46-4299-9522-ac835c0adc61\") " pod="openshift-image-registry/node-ca-k7mgr"
Sep 17 17:57:02.099390 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.098920    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-host-var-lib-cni-multus\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.099390 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.098939    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-run-openvswitch\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.099390 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.098961    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-host-var-lib-cni-bin\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.099390 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.098982    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-etc-kubernetes\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.099390 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099005    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-host-cni-netd\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.099390 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099024    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-etc-sysctl-d\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:02.099390 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-multus-socket-dir-parent\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.099390 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099071    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6345bbfb-99aa-42ba-b5c9-d7c46ef2392f-hosts-file\") pod \"node-resolver-2wq8v\" (UID: \"6345bbfb-99aa-42ba-b5c9-d7c46ef2392f\") " pod="openshift-dns/node-resolver-2wq8v"
Sep 17 17:57:02.099390 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099093    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-host-kubelet\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.099390 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099115    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-etc-kubernetes\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:02.099390 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099144    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-multus-conf-dir\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.099390 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099209    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-host-run-ovn-kubernetes\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.099390 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-host-cni-bin\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.099390 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099255    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-system-cni-dir\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.099390 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099261    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6345bbfb-99aa-42ba-b5c9-d7c46ef2392f-tmp-dir\") pod \"node-resolver-2wq8v\" (UID: \"6345bbfb-99aa-42ba-b5c9-d7c46ef2392f\") " pod="openshift-dns/node-resolver-2wq8v"
Sep 17 17:57:02.099390 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099281    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-run-systemd\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.099390 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099316    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-etc-sysconfig\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:02.100054 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099321    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-run-systemd\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.100054 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099340    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-sys\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:02.100054 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099362    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ba4edf3-c791-4134-9ea3-449a5c3b437c-tmp\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:02.100054 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099365    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/383b592e-5a46-4299-9522-ac835c0adc61-host\") pod \"node-ca-k7mgr\" (UID: \"383b592e-5a46-4299-9522-ac835c0adc61\") " pod="openshift-image-registry/node-ca-k7mgr"
Sep 17 17:57:02.100054 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099386    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/41ab02fa-694e-4b41-8969-0e5181664b3a-kubelet-config\") pod \"global-pull-secret-syncer-shvtg\" (UID: \"41ab02fa-694e-4b41-8969-0e5181664b3a\") " pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:02.100054 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099400    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-run-openvswitch\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.100054 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099416    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-ovn-node-metrics-cert\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.100054 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099428    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd3ffd9f-338b-48ca-97c2-4292e8716182-multus-daemon-config\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.100054 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099440    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-etc-systemd\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:02.100054 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099443    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-host-var-lib-cni-bin\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.100054 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.098680    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-lib-modules\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:02.100054 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099503    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-etc-kubernetes\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.100054 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099527    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b4f2a63c-9017-431f-bec9-47521af4d1bd-host-slash\") pod \"iptables-alerter-4qkk8\" (UID: \"b4f2a63c-9017-431f-bec9-47521af4d1bd\") " pod="openshift-network-operator/iptables-alerter-4qkk8"
Sep 17 17:57:02.100054 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099556    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-node-log\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.100054 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099576    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njd8d\" (UniqueName: \"kubernetes.io/projected/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-kube-api-access-njd8d\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.100054 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099579    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-etc-sysctl-d\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:02.100054 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099677    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-run\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:02.100054 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099816    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/41ab02fa-694e-4b41-8969-0e5181664b3a-dbus\") pod \"global-pull-secret-syncer-shvtg\" (UID: \"41ab02fa-694e-4b41-8969-0e5181664b3a\") " pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:02.100736 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099827    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-multus-socket-dir-parent\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.100736 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099864    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-sys\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:02.100736 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099878    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6345bbfb-99aa-42ba-b5c9-d7c46ef2392f-hosts-file\") pod \"node-resolver-2wq8v\" (UID: \"6345bbfb-99aa-42ba-b5c9-d7c46ef2392f\") " pod="openshift-dns/node-resolver-2wq8v"
Sep 17 17:57:02.100736 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099897    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-host-cni-netd\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.100736 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099913    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-host-kubelet\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.100736 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099934    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-etc-openvswitch\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.100736 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099955    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-etc-kubernetes\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:02.100736 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099971    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-node-log\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.100736 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.099995    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-multus-conf-dir\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.100736 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.100034    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-host-run-ovn-kubernetes\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.100736 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.100190    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-host-cni-bin\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.100736 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.100173    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-etc-systemd\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:02.100736 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.100282    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-etc-sysconfig\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:02.100736 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.100305    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-system-cni-dir\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.100736 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.100330    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ca53e15c-51d9-40c8-9d87-80e217ce1815-konnectivity-ca\") pod \"konnectivity-agent-sf7g4\" (UID: \"ca53e15c-51d9-40c8-9d87-80e217ce1815\") " pod="kube-system/konnectivity-agent-sf7g4"
Sep 17 17:57:02.100736 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.100348    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/41ab02fa-694e-4b41-8969-0e5181664b3a-kubelet-config\") pod \"global-pull-secret-syncer-shvtg\" (UID: \"41ab02fa-694e-4b41-8969-0e5181664b3a\") " pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:02.100736 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.100366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-multus-cni-dir\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.100736 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.100389    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzq5m\" (UniqueName: \"kubernetes.io/projected/bd3ffd9f-338b-48ca-97c2-4292e8716182-kube-api-access-vzq5m\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.101332 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.100412    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-etc-modprobe-d\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:02.101332 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.100433    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b4f2a63c-9017-431f-bec9-47521af4d1bd-iptables-alerter-script\") pod \"iptables-alerter-4qkk8\" (UID: \"b4f2a63c-9017-431f-bec9-47521af4d1bd\") " pod="openshift-network-operator/iptables-alerter-4qkk8"
Sep 17 17:57:02.101332 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.100455    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41ab02fa-694e-4b41-8969-0e5181664b3a-original-pull-secret\") pod \"global-pull-secret-syncer-shvtg\" (UID: \"41ab02fa-694e-4b41-8969-0e5181664b3a\") " pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:02.101332 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.100476    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/383b592e-5a46-4299-9522-ac835c0adc61-serviceca\") pod \"node-ca-k7mgr\" (UID: \"383b592e-5a46-4299-9522-ac835c0adc61\") " pod="openshift-image-registry/node-ca-k7mgr"
Sep 17 17:57:02.101332 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.100499    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.101332 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.100522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-ovnkube-config\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.101332 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.100546    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7ba4edf3-c791-4134-9ea3-449a5c3b437c-etc-tuned\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:02.101332 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.100685    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-host-run-netns\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.101332 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.100766    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-host-slash\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.101332 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.100791    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-host-run-netns\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.101332 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.100813    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ca53e15c-51d9-40c8-9d87-80e217ce1815-agent-certs\") pod \"konnectivity-agent-sf7g4\" (UID: \"ca53e15c-51d9-40c8-9d87-80e217ce1815\") " pod="kube-system/konnectivity-agent-sf7g4"
Sep 17 17:57:02.101332 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.100836    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-etc-sysctl-conf\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:02.101332 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.100842    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ca53e15c-51d9-40c8-9d87-80e217ce1815-konnectivity-ca\") pod \"konnectivity-agent-sf7g4\" (UID: \"ca53e15c-51d9-40c8-9d87-80e217ce1815\") " pod="kube-system/konnectivity-agent-sf7g4"
Sep 17 17:57:02.101332 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.100860    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfc5w\" (UniqueName: \"kubernetes.io/projected/7ba4edf3-c791-4134-9ea3-449a5c3b437c-kube-api-access-wfc5w\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:02.101332 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.100971    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b4f2a63c-9017-431f-bec9-47521af4d1bd-iptables-alerter-script\") pod \"iptables-alerter-4qkk8\" (UID: \"b4f2a63c-9017-431f-bec9-47521af4d1bd\") " pod="openshift-network-operator/iptables-alerter-4qkk8"
Sep 17 17:57:02.101332 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.101025    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-multus-cni-dir\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.101332 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.101114    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-host-run-netns\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.102230 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.101184    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-host-slash\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.102230 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.101232    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-host-run-netns\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.102230 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.101301    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-etc-modprobe-d\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:02.102230 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:02.101510    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:02.102230 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:02.101577    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41ab02fa-694e-4b41-8969-0e5181664b3a-original-pull-secret podName:41ab02fa-694e-4b41-8969-0e5181664b3a nodeName:}" failed. No retries permitted until 2026-09-17 17:57:02.601560286 +0000 UTC m=+4.702024083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41ab02fa-694e-4b41-8969-0e5181664b3a-original-pull-secret") pod "global-pull-secret-syncer-shvtg" (UID: "41ab02fa-694e-4b41-8969-0e5181664b3a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:02.102230 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.101592    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-etc-sysctl-conf\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:02.102230 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.101762    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-host-var-lib-kubelet\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.102230 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.101796    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-run-ovn\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.102230 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.101820    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-env-overrides\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.102230 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.101841    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-var-lib-kubelet\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:02.102230 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.101861    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd3ffd9f-338b-48ca-97c2-4292e8716182-cni-binary-copy\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.102230 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.101883    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-host-run-multus-certs\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.102230 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.101904    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-ovnkube-script-lib\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.102230 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.101924    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-host-run-k8s-cni-cncf-io\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.102230 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.101944    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqhgj\" (UniqueName: \"kubernetes.io/projected/b4f2a63c-9017-431f-bec9-47521af4d1bd-kube-api-access-xqhgj\") pod \"iptables-alerter-4qkk8\" (UID: \"b4f2a63c-9017-431f-bec9-47521af4d1bd\") " pod="openshift-network-operator/iptables-alerter-4qkk8"
Sep 17 17:57:02.102230 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.102091    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ba4edf3-c791-4134-9ea3-449a5c3b437c-var-lib-kubelet\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:02.102230 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.102114    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-ovnkube-config\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.103017 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.102141    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-host-var-lib-kubelet\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.103017 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.102209    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.103017 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.102221    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-host-run-k8s-cni-cncf-io\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.103017 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.102253    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-run-ovn\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.103017 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.102261    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd3ffd9f-338b-48ca-97c2-4292e8716182-host-run-multus-certs\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.103017 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.102541    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-env-overrides\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.103017 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.102675    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd3ffd9f-338b-48ca-97c2-4292e8716182-cni-binary-copy\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.103017 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.102664    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/383b592e-5a46-4299-9522-ac835c0adc61-serviceca\") pod \"node-ca-k7mgr\" (UID: \"383b592e-5a46-4299-9522-ac835c0adc61\") " pod="openshift-image-registry/node-ca-k7mgr"
Sep 17 17:57:02.103017 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.102995    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-ovnkube-script-lib\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.103463 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.103227    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ba4edf3-c791-4134-9ea3-449a5c3b437c-tmp\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:02.103463 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.103263    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7ba4edf3-c791-4134-9ea3-449a5c3b437c-etc-tuned\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:02.103463 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.103323    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-ovn-node-metrics-cert\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.103891 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.103875    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ca53e15c-51d9-40c8-9d87-80e217ce1815-agent-certs\") pod \"konnectivity-agent-sf7g4\" (UID: \"ca53e15c-51d9-40c8-9d87-80e217ce1815\") " pod="kube-system/konnectivity-agent-sf7g4"
Sep 17 17:57:02.117198 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.117175    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jsgj\" (UniqueName: \"kubernetes.io/projected/383b592e-5a46-4299-9522-ac835c0adc61-kube-api-access-6jsgj\") pod \"node-ca-k7mgr\" (UID: \"383b592e-5a46-4299-9522-ac835c0adc61\") " pod="openshift-image-registry/node-ca-k7mgr"
Sep 17 17:57:02.118113 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.118090    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfc5w\" (UniqueName: \"kubernetes.io/projected/7ba4edf3-c791-4134-9ea3-449a5c3b437c-kube-api-access-wfc5w\") pod \"tuned-mvn46\" (UID: \"7ba4edf3-c791-4134-9ea3-449a5c3b437c\") " pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:02.118292 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.118272    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqhgj\" (UniqueName: \"kubernetes.io/projected/b4f2a63c-9017-431f-bec9-47521af4d1bd-kube-api-access-xqhgj\") pod \"iptables-alerter-4qkk8\" (UID: \"b4f2a63c-9017-431f-bec9-47521af4d1bd\") " pod="openshift-network-operator/iptables-alerter-4qkk8"
Sep 17 17:57:02.120590 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.120543    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8kfc\" (UniqueName: \"kubernetes.io/projected/6345bbfb-99aa-42ba-b5c9-d7c46ef2392f-kube-api-access-r8kfc\") pod \"node-resolver-2wq8v\" (UID: \"6345bbfb-99aa-42ba-b5c9-d7c46ef2392f\") " pod="openshift-dns/node-resolver-2wq8v"
Sep 17 17:57:02.120829 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.120811    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njd8d\" (UniqueName: \"kubernetes.io/projected/913ce1e9-9fb6-4049-a06d-9ebc69d3e528-kube-api-access-njd8d\") pod \"ovnkube-node-qm8hx\" (UID: \"913ce1e9-9fb6-4049-a06d-9ebc69d3e528\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.127417 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.127397    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzq5m\" (UniqueName: \"kubernetes.io/projected/bd3ffd9f-338b-48ca-97c2-4292e8716182-kube-api-access-vzq5m\") pod \"multus-xhbcn\" (UID: \"bd3ffd9f-338b-48ca-97c2-4292e8716182\") " pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.199220 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.199186    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8qdrq"
Sep 17 17:57:02.208633 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.208609    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7czlw"
Sep 17 17:57:02.217736 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.217713    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k7mgr"
Sep 17 17:57:02.224627 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.224607    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4qkk8"
Sep 17 17:57:02.231250 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.231224    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2wq8v"
Sep 17 17:57:02.236990 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.236971    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:02.244499 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.244482    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sf7g4"
Sep 17 17:57:02.248946 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.248930    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mvn46"
Sep 17 17:57:02.253663 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.253650    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xhbcn"
Sep 17 17:57:02.336273 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.336246    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:57:02.504510 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.504434    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11b6807f-cb26-4117-b06d-eb974e5640ae-metrics-certs\") pod \"network-metrics-daemon-dsz4t\" (UID: \"11b6807f-cb26-4117-b06d-eb974e5640ae\") " pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:57:02.504661 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:02.504606    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:02.504721 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:02.504677    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11b6807f-cb26-4117-b06d-eb974e5640ae-metrics-certs podName:11b6807f-cb26-4117-b06d-eb974e5640ae nodeName:}" failed. No retries permitted until 2026-09-17 17:57:03.504657427 +0000 UTC m=+5.605121242 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11b6807f-cb26-4117-b06d-eb974e5640ae-metrics-certs") pod "network-metrics-daemon-dsz4t" (UID: "11b6807f-cb26-4117-b06d-eb974e5640ae") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:02.605383 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.605340    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41ab02fa-694e-4b41-8969-0e5181664b3a-original-pull-secret\") pod \"global-pull-secret-syncer-shvtg\" (UID: \"41ab02fa-694e-4b41-8969-0e5181664b3a\") " pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:02.605383 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.605384    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2hn2\" (UniqueName: \"kubernetes.io/projected/c53ddf27-1765-4785-b122-1645ccc8d094-kube-api-access-p2hn2\") pod \"network-check-target-98c4r\" (UID: \"c53ddf27-1765-4785-b122-1645ccc8d094\") " pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:57:02.605600 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:02.605484    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:02.605600 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:02.605496    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:57:02.605600 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:02.605515    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:57:02.605600 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:02.605528    2581 projected.go:196] Error preparing data for projected volume kube-api-access-p2hn2 for pod openshift-network-diagnostics/network-check-target-98c4r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:02.605600 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:02.605550    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41ab02fa-694e-4b41-8969-0e5181664b3a-original-pull-secret podName:41ab02fa-694e-4b41-8969-0e5181664b3a nodeName:}" failed. No retries permitted until 2026-09-17 17:57:03.605531987 +0000 UTC m=+5.705995802 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41ab02fa-694e-4b41-8969-0e5181664b3a-original-pull-secret") pod "global-pull-secret-syncer-shvtg" (UID: "41ab02fa-694e-4b41-8969-0e5181664b3a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:02.605600 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:02.605571    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c53ddf27-1765-4785-b122-1645ccc8d094-kube-api-access-p2hn2 podName:c53ddf27-1765-4785-b122-1645ccc8d094 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:03.605561117 +0000 UTC m=+5.706024932 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-p2hn2" (UniqueName: "kubernetes.io/projected/c53ddf27-1765-4785-b122-1645ccc8d094-kube-api-access-p2hn2") pod "network-check-target-98c4r" (UID: "c53ddf27-1765-4785-b122-1645ccc8d094") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:02.916813 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:57:02.916708    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f2a63c_9017_431f_bec9_47521af4d1bd.slice/crio-485e6b21c6592a3bd2d14ebbfeee4ee94df9dd83eae0060d668fc719b6320edd WatchSource:0}: Error finding container 485e6b21c6592a3bd2d14ebbfeee4ee94df9dd83eae0060d668fc719b6320edd: Status 404 returned error can't find the container with id 485e6b21c6592a3bd2d14ebbfeee4ee94df9dd83eae0060d668fc719b6320edd
Sep 17 17:57:02.917946 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:57:02.917925    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod383b592e_5a46_4299_9522_ac835c0adc61.slice/crio-485068e7cb8dc6670edef8cddd6bc5155ab137c76dde8f285112b7a798851c3e WatchSource:0}: Error finding container 485068e7cb8dc6670edef8cddd6bc5155ab137c76dde8f285112b7a798851c3e: Status 404 returned error can't find the container with id 485068e7cb8dc6670edef8cddd6bc5155ab137c76dde8f285112b7a798851c3e
Sep 17 17:57:02.918538 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:57:02.918508    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca53e15c_51d9_40c8_9d87_80e217ce1815.slice/crio-6e260ac5732d8e18320106797d96d80dd3fa1ee5d481233e50f8ae74b2616a7b WatchSource:0}: Error finding container 6e260ac5732d8e18320106797d96d80dd3fa1ee5d481233e50f8ae74b2616a7b: Status 404 returned error can't find the container with id 6e260ac5732d8e18320106797d96d80dd3fa1ee5d481233e50f8ae74b2616a7b
Sep 17 17:57:02.919983 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:57:02.919961    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3ffd9f_338b_48ca_97c2_4292e8716182.slice/crio-229f0e7d93ae269cf5a913df076f7f42fa340fcaddd19ed4a247eca2e34964e2 WatchSource:0}: Error finding container 229f0e7d93ae269cf5a913df076f7f42fa340fcaddd19ed4a247eca2e34964e2: Status 404 returned error can't find the container with id 229f0e7d93ae269cf5a913df076f7f42fa340fcaddd19ed4a247eca2e34964e2
Sep 17 17:57:02.922773 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:57:02.922747    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba4edf3_c791_4134_9ea3_449a5c3b437c.slice/crio-d19cb709444ab5f0f650492b8aa3b8bb68a42d2a3a6202407d725fe57327a631 WatchSource:0}: Error finding container d19cb709444ab5f0f650492b8aa3b8bb68a42d2a3a6202407d725fe57327a631: Status 404 returned error can't find the container with id d19cb709444ab5f0f650492b8aa3b8bb68a42d2a3a6202407d725fe57327a631
Sep 17 17:57:02.925181 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:57:02.925117    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb24cd0_f80d_426b_9910_71b2b1a4d578.slice/crio-15b5189470d8fad8a7f1fb057ab31da310c98727a5a0bb165e07f9846e188d2f WatchSource:0}: Error finding container 15b5189470d8fad8a7f1fb057ab31da310c98727a5a0bb165e07f9846e188d2f: Status 404 returned error can't find the container with id 15b5189470d8fad8a7f1fb057ab31da310c98727a5a0bb165e07f9846e188d2f
Sep 17 17:57:02.926433 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:57:02.926416    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38076ecb_73ae_4c1c_8359_b9e2b7435a7b.slice/crio-643218054da455d7258e127dff62128926653628473b89a11c1956b588db2127 WatchSource:0}: Error finding container 643218054da455d7258e127dff62128926653628473b89a11c1956b588db2127: Status 404 returned error can't find the container with id 643218054da455d7258e127dff62128926653628473b89a11c1956b588db2127
Sep 17 17:57:02.927513 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:57:02.927472    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod913ce1e9_9fb6_4049_a06d_9ebc69d3e528.slice/crio-4cf650c7465eb6772d60ef82a96595abb26f72ca536781c5c08ac320a84015d3 WatchSource:0}: Error finding container 4cf650c7465eb6772d60ef82a96595abb26f72ca536781c5c08ac320a84015d3: Status 404 returned error can't find the container with id 4cf650c7465eb6772d60ef82a96595abb26f72ca536781c5c08ac320a84015d3
Sep 17 17:57:02.960878 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.960782    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:52:01 +0000 UTC" deadline="2028-04-18 21:06:14.299603773 +0000 UTC"
Sep 17 17:57:02.960878 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:02.960838    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13899h9m11.338770262s"
Sep 17 17:57:03.052729 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:03.052683    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-209.ec2.internal" event={"ID":"284822b4778f9378098f5e86308878b3","Type":"ContainerStarted","Data":"ad64f417dbdf20665a652092b207c7a0d7d1823835243654c170069876a0e67f"}
Sep 17 17:57:03.055240 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:03.055182    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2wq8v" event={"ID":"6345bbfb-99aa-42ba-b5c9-d7c46ef2392f","Type":"ContainerStarted","Data":"18e312c4f181f41ed0ee0ecf9b038edcb00609f68c603cc0cdffd272fffad1c8"}
Sep 17 17:57:03.057342 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:03.057315    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx" event={"ID":"913ce1e9-9fb6-4049-a06d-9ebc69d3e528","Type":"ContainerStarted","Data":"4cf650c7465eb6772d60ef82a96595abb26f72ca536781c5c08ac320a84015d3"}
Sep 17 17:57:03.058434 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:03.058409    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k7mgr" event={"ID":"383b592e-5a46-4299-9522-ac835c0adc61","Type":"ContainerStarted","Data":"485068e7cb8dc6670edef8cddd6bc5155ab137c76dde8f285112b7a798851c3e"}
Sep 17 17:57:03.060423 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:03.060390    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4qkk8" event={"ID":"b4f2a63c-9017-431f-bec9-47521af4d1bd","Type":"ContainerStarted","Data":"485e6b21c6592a3bd2d14ebbfeee4ee94df9dd83eae0060d668fc719b6320edd"}
Sep 17 17:57:03.061496 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:03.061477    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8qdrq" event={"ID":"38076ecb-73ae-4c1c-8359-b9e2b7435a7b","Type":"ContainerStarted","Data":"643218054da455d7258e127dff62128926653628473b89a11c1956b588db2127"}
Sep 17 17:57:03.062513 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:03.062493    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7czlw" event={"ID":"ddb24cd0-f80d-426b-9910-71b2b1a4d578","Type":"ContainerStarted","Data":"15b5189470d8fad8a7f1fb057ab31da310c98727a5a0bb165e07f9846e188d2f"}
Sep 17 17:57:03.064689 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:03.064671    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mvn46" event={"ID":"7ba4edf3-c791-4134-9ea3-449a5c3b437c","Type":"ContainerStarted","Data":"d19cb709444ab5f0f650492b8aa3b8bb68a42d2a3a6202407d725fe57327a631"}
Sep 17 17:57:03.067062 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:03.067045    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xhbcn" event={"ID":"bd3ffd9f-338b-48ca-97c2-4292e8716182","Type":"ContainerStarted","Data":"229f0e7d93ae269cf5a913df076f7f42fa340fcaddd19ed4a247eca2e34964e2"}
Sep 17 17:57:03.068077 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:03.068059    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sf7g4" event={"ID":"ca53e15c-51d9-40c8-9d87-80e217ce1815","Type":"ContainerStarted","Data":"6e260ac5732d8e18320106797d96d80dd3fa1ee5d481233e50f8ae74b2616a7b"}
Sep 17 17:57:03.068552 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:03.068515    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-209.ec2.internal" podStartSLOduration=2.068503917 podStartE2EDuration="2.068503917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:03.068236695 +0000 UTC m=+5.168700513" watchObservedRunningTime="2026-09-17 17:57:03.068503917 +0000 UTC m=+5.168967736"
Sep 17 17:57:03.512799 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:03.512726    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11b6807f-cb26-4117-b06d-eb974e5640ae-metrics-certs\") pod \"network-metrics-daemon-dsz4t\" (UID: \"11b6807f-cb26-4117-b06d-eb974e5640ae\") " pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:57:03.512956 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:03.512876    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:03.512956 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:03.512936    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11b6807f-cb26-4117-b06d-eb974e5640ae-metrics-certs podName:11b6807f-cb26-4117-b06d-eb974e5640ae nodeName:}" failed. No retries permitted until 2026-09-17 17:57:05.512917892 +0000 UTC m=+7.613381691 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11b6807f-cb26-4117-b06d-eb974e5640ae-metrics-certs") pod "network-metrics-daemon-dsz4t" (UID: "11b6807f-cb26-4117-b06d-eb974e5640ae") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:03.614295 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:03.613586    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41ab02fa-694e-4b41-8969-0e5181664b3a-original-pull-secret\") pod \"global-pull-secret-syncer-shvtg\" (UID: \"41ab02fa-694e-4b41-8969-0e5181664b3a\") " pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:03.614295 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:03.613632    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2hn2\" (UniqueName: \"kubernetes.io/projected/c53ddf27-1765-4785-b122-1645ccc8d094-kube-api-access-p2hn2\") pod \"network-check-target-98c4r\" (UID: \"c53ddf27-1765-4785-b122-1645ccc8d094\") " pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:57:03.614295 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:03.613758    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:57:03.614295 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:03.613773    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:57:03.614295 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:03.613785    2581 projected.go:196] Error preparing data for projected volume kube-api-access-p2hn2 for pod openshift-network-diagnostics/network-check-target-98c4r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:03.614295 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:03.613835    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c53ddf27-1765-4785-b122-1645ccc8d094-kube-api-access-p2hn2 podName:c53ddf27-1765-4785-b122-1645ccc8d094 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:05.613819056 +0000 UTC m=+7.714282859 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-p2hn2" (UniqueName: "kubernetes.io/projected/c53ddf27-1765-4785-b122-1645ccc8d094-kube-api-access-p2hn2") pod "network-check-target-98c4r" (UID: "c53ddf27-1765-4785-b122-1645ccc8d094") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:03.614295 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:03.614213    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:03.614295 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:03.614262    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41ab02fa-694e-4b41-8969-0e5181664b3a-original-pull-secret podName:41ab02fa-694e-4b41-8969-0e5181664b3a nodeName:}" failed. No retries permitted until 2026-09-17 17:57:05.61424792 +0000 UTC m=+7.714711720 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41ab02fa-694e-4b41-8969-0e5181664b3a-original-pull-secret") pod "global-pull-secret-syncer-shvtg" (UID: "41ab02fa-694e-4b41-8969-0e5181664b3a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:03.961330 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:03.961185    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:52:01 +0000 UTC" deadline="2028-03-26 14:08:34.234878304 +0000 UTC"
Sep 17 17:57:03.961330 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:03.961232    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13340h11m30.273651059s"
Sep 17 17:57:04.044764 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:04.044266    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:57:04.044764 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:04.044395    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98c4r" podUID="c53ddf27-1765-4785-b122-1645ccc8d094"
Sep 17 17:57:04.044764 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:04.044420    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:57:04.044764 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:04.044543    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsz4t" podUID="11b6807f-cb26-4117-b06d-eb974e5640ae"
Sep 17 17:57:04.044764 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:04.044614    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:04.044764 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:04.044692    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shvtg" podUID="41ab02fa-694e-4b41-8969-0e5181664b3a"
Sep 17 17:57:04.086845 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:04.086810    2581 generic.go:356] "Generic (PLEG): container finished" podID="c819eb604d67aa7e331f4a25bba966ae" containerID="8f0c5c722bb461c34fbfd4e596fc93826cbb0f534b6340ba584955dd1fbc90ef" exitCode=0
Sep 17 17:57:04.087396 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:04.087372    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-209.ec2.internal" event={"ID":"c819eb604d67aa7e331f4a25bba966ae","Type":"ContainerDied","Data":"8f0c5c722bb461c34fbfd4e596fc93826cbb0f534b6340ba584955dd1fbc90ef"}
Sep 17 17:57:05.107337 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:05.107297    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-209.ec2.internal" event={"ID":"c819eb604d67aa7e331f4a25bba966ae","Type":"ContainerStarted","Data":"95830f40e3aab3657ec97805ab0b9f79020450abb7e2ca7c7f02a47b04f9089e"}
Sep 17 17:57:05.534079 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:05.533985    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11b6807f-cb26-4117-b06d-eb974e5640ae-metrics-certs\") pod \"network-metrics-daemon-dsz4t\" (UID: \"11b6807f-cb26-4117-b06d-eb974e5640ae\") " pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:57:05.534247 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:05.534187    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:05.534307 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:05.534257    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11b6807f-cb26-4117-b06d-eb974e5640ae-metrics-certs podName:11b6807f-cb26-4117-b06d-eb974e5640ae nodeName:}" failed. No retries permitted until 2026-09-17 17:57:09.534237134 +0000 UTC m=+11.634700938 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11b6807f-cb26-4117-b06d-eb974e5640ae-metrics-certs") pod "network-metrics-daemon-dsz4t" (UID: "11b6807f-cb26-4117-b06d-eb974e5640ae") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:05.634556 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:05.634516    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41ab02fa-694e-4b41-8969-0e5181664b3a-original-pull-secret\") pod \"global-pull-secret-syncer-shvtg\" (UID: \"41ab02fa-694e-4b41-8969-0e5181664b3a\") " pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:05.634730 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:05.634572    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2hn2\" (UniqueName: \"kubernetes.io/projected/c53ddf27-1765-4785-b122-1645ccc8d094-kube-api-access-p2hn2\") pod \"network-check-target-98c4r\" (UID: \"c53ddf27-1765-4785-b122-1645ccc8d094\") " pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:57:05.634788 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:05.634754    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:57:05.634788 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:05.634773    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:57:05.634788 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:05.634785    2581 projected.go:196] Error preparing data for projected volume kube-api-access-p2hn2 for pod openshift-network-diagnostics/network-check-target-98c4r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:05.634938 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:05.634842    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c53ddf27-1765-4785-b122-1645ccc8d094-kube-api-access-p2hn2 podName:c53ddf27-1765-4785-b122-1645ccc8d094 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:09.634823699 +0000 UTC m=+11.735287516 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-p2hn2" (UniqueName: "kubernetes.io/projected/c53ddf27-1765-4785-b122-1645ccc8d094-kube-api-access-p2hn2") pod "network-check-target-98c4r" (UID: "c53ddf27-1765-4785-b122-1645ccc8d094") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:05.635250 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:05.635230    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:05.635355 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:05.635281    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41ab02fa-694e-4b41-8969-0e5181664b3a-original-pull-secret podName:41ab02fa-694e-4b41-8969-0e5181664b3a nodeName:}" failed. No retries permitted until 2026-09-17 17:57:09.635267776 +0000 UTC m=+11.735731576 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41ab02fa-694e-4b41-8969-0e5181664b3a-original-pull-secret") pod "global-pull-secret-syncer-shvtg" (UID: "41ab02fa-694e-4b41-8969-0e5181664b3a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:06.045360 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:06.045281    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:57:06.045518 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:06.045412    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98c4r" podUID="c53ddf27-1765-4785-b122-1645ccc8d094"
Sep 17 17:57:06.045832 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:06.045617    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:06.045832 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:06.045666    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:57:06.045832 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:06.045782    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shvtg" podUID="41ab02fa-694e-4b41-8969-0e5181664b3a"
Sep 17 17:57:06.046022 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:06.045857    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsz4t" podUID="11b6807f-cb26-4117-b06d-eb974e5640ae"
Sep 17 17:57:08.046034 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:08.044248    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:57:08.046034 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:08.044370    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98c4r" podUID="c53ddf27-1765-4785-b122-1645ccc8d094"
Sep 17 17:57:08.046034 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:08.044734    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:57:08.046034 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:08.044832    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsz4t" podUID="11b6807f-cb26-4117-b06d-eb974e5640ae"
Sep 17 17:57:08.046034 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:08.044979    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:08.046034 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:08.045171    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shvtg" podUID="41ab02fa-694e-4b41-8969-0e5181664b3a"
Sep 17 17:57:09.568534 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:09.567989    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11b6807f-cb26-4117-b06d-eb974e5640ae-metrics-certs\") pod \"network-metrics-daemon-dsz4t\" (UID: \"11b6807f-cb26-4117-b06d-eb974e5640ae\") " pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:57:09.568534 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:09.568131    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:09.568534 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:09.568208    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11b6807f-cb26-4117-b06d-eb974e5640ae-metrics-certs podName:11b6807f-cb26-4117-b06d-eb974e5640ae nodeName:}" failed. No retries permitted until 2026-09-17 17:57:17.568189731 +0000 UTC m=+19.668653532 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11b6807f-cb26-4117-b06d-eb974e5640ae-metrics-certs") pod "network-metrics-daemon-dsz4t" (UID: "11b6807f-cb26-4117-b06d-eb974e5640ae") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:09.668843 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:09.668806    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41ab02fa-694e-4b41-8969-0e5181664b3a-original-pull-secret\") pod \"global-pull-secret-syncer-shvtg\" (UID: \"41ab02fa-694e-4b41-8969-0e5181664b3a\") " pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:09.669001 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:09.668855    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2hn2\" (UniqueName: \"kubernetes.io/projected/c53ddf27-1765-4785-b122-1645ccc8d094-kube-api-access-p2hn2\") pod \"network-check-target-98c4r\" (UID: \"c53ddf27-1765-4785-b122-1645ccc8d094\") " pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:57:09.669001 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:09.668969    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:57:09.669001 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:09.668986    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:57:09.669001 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:09.668997    2581 projected.go:196] Error preparing data for projected volume kube-api-access-p2hn2 for pod openshift-network-diagnostics/network-check-target-98c4r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:09.669180 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:09.669048    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c53ddf27-1765-4785-b122-1645ccc8d094-kube-api-access-p2hn2 podName:c53ddf27-1765-4785-b122-1645ccc8d094 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:17.669030324 +0000 UTC m=+19.769494141 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-p2hn2" (UniqueName: "kubernetes.io/projected/c53ddf27-1765-4785-b122-1645ccc8d094-kube-api-access-p2hn2") pod "network-check-target-98c4r" (UID: "c53ddf27-1765-4785-b122-1645ccc8d094") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:09.669410 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:09.669395    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:09.669457 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:09.669433    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41ab02fa-694e-4b41-8969-0e5181664b3a-original-pull-secret podName:41ab02fa-694e-4b41-8969-0e5181664b3a nodeName:}" failed. No retries permitted until 2026-09-17 17:57:17.669424068 +0000 UTC m=+19.769887865 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41ab02fa-694e-4b41-8969-0e5181664b3a-original-pull-secret") pod "global-pull-secret-syncer-shvtg" (UID: "41ab02fa-694e-4b41-8969-0e5181664b3a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:10.044905 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:10.044830    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:57:10.045061 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:10.044925    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsz4t" podUID="11b6807f-cb26-4117-b06d-eb974e5640ae"
Sep 17 17:57:10.045270 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:10.045254    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:10.045337 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:10.045319    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shvtg" podUID="41ab02fa-694e-4b41-8969-0e5181664b3a"
Sep 17 17:57:10.045438 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:10.045428    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:57:10.045486 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:10.045474    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98c4r" podUID="c53ddf27-1765-4785-b122-1645ccc8d094"
Sep 17 17:57:12.043559 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:12.043522    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:57:12.044118 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:12.043670    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:12.044118 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:12.043697    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsz4t" podUID="11b6807f-cb26-4117-b06d-eb974e5640ae"
Sep 17 17:57:12.044118 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:12.043755    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:57:12.044118 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:12.043749    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shvtg" podUID="41ab02fa-694e-4b41-8969-0e5181664b3a"
Sep 17 17:57:12.044118 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:12.043823    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98c4r" podUID="c53ddf27-1765-4785-b122-1645ccc8d094"
Sep 17 17:57:14.044124 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:14.044087    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:57:14.044124 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:14.044095    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:14.044614 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:14.044263    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsz4t" podUID="11b6807f-cb26-4117-b06d-eb974e5640ae"
Sep 17 17:57:14.044614 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:14.044308    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:57:14.044614 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:14.044403    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shvtg" podUID="41ab02fa-694e-4b41-8969-0e5181664b3a"
Sep 17 17:57:14.044614 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:14.044517    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98c4r" podUID="c53ddf27-1765-4785-b122-1645ccc8d094"
Sep 17 17:57:16.043449 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:16.043414    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:57:16.043881 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:16.043414    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:16.043881 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:16.043532    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsz4t" podUID="11b6807f-cb26-4117-b06d-eb974e5640ae"
Sep 17 17:57:16.043881 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:16.043657    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shvtg" podUID="41ab02fa-694e-4b41-8969-0e5181664b3a"
Sep 17 17:57:16.043881 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:16.043701    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:57:16.043881 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:16.043829    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98c4r" podUID="c53ddf27-1765-4785-b122-1645ccc8d094"
Sep 17 17:57:17.628331 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:17.628283    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11b6807f-cb26-4117-b06d-eb974e5640ae-metrics-certs\") pod \"network-metrics-daemon-dsz4t\" (UID: \"11b6807f-cb26-4117-b06d-eb974e5640ae\") " pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:57:17.628807 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:17.628420    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:17.628807 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:17.628490    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11b6807f-cb26-4117-b06d-eb974e5640ae-metrics-certs podName:11b6807f-cb26-4117-b06d-eb974e5640ae nodeName:}" failed. No retries permitted until 2026-09-17 17:57:33.628471716 +0000 UTC m=+35.728935538 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11b6807f-cb26-4117-b06d-eb974e5640ae-metrics-certs") pod "network-metrics-daemon-dsz4t" (UID: "11b6807f-cb26-4117-b06d-eb974e5640ae") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:17.729390 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:17.729352    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41ab02fa-694e-4b41-8969-0e5181664b3a-original-pull-secret\") pod \"global-pull-secret-syncer-shvtg\" (UID: \"41ab02fa-694e-4b41-8969-0e5181664b3a\") " pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:17.729390 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:17.729402    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2hn2\" (UniqueName: \"kubernetes.io/projected/c53ddf27-1765-4785-b122-1645ccc8d094-kube-api-access-p2hn2\") pod \"network-check-target-98c4r\" (UID: \"c53ddf27-1765-4785-b122-1645ccc8d094\") " pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:57:17.729641 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:17.729517    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:17.729641 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:17.729538    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:57:17.729641 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:17.729557    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:57:17.729641 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:17.729568    2581 projected.go:196] Error preparing data for projected volume kube-api-access-p2hn2 for pod openshift-network-diagnostics/network-check-target-98c4r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:17.729641 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:17.729580    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41ab02fa-694e-4b41-8969-0e5181664b3a-original-pull-secret podName:41ab02fa-694e-4b41-8969-0e5181664b3a nodeName:}" failed. No retries permitted until 2026-09-17 17:57:33.729562577 +0000 UTC m=+35.830026390 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41ab02fa-694e-4b41-8969-0e5181664b3a-original-pull-secret") pod "global-pull-secret-syncer-shvtg" (UID: "41ab02fa-694e-4b41-8969-0e5181664b3a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:17.729641 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:17.729601    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c53ddf27-1765-4785-b122-1645ccc8d094-kube-api-access-p2hn2 podName:c53ddf27-1765-4785-b122-1645ccc8d094 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:33.729590438 +0000 UTC m=+35.830054235 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-p2hn2" (UniqueName: "kubernetes.io/projected/c53ddf27-1765-4785-b122-1645ccc8d094-kube-api-access-p2hn2") pod "network-check-target-98c4r" (UID: "c53ddf27-1765-4785-b122-1645ccc8d094") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:18.043986 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:18.043909    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:57:18.044189 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:18.044044    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98c4r" podUID="c53ddf27-1765-4785-b122-1645ccc8d094"
Sep 17 17:57:18.044501 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:18.044485    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:57:18.044613 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:18.044588    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsz4t" podUID="11b6807f-cb26-4117-b06d-eb974e5640ae"
Sep 17 17:57:18.044759 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:18.044739    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:18.044860 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:18.044837    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shvtg" podUID="41ab02fa-694e-4b41-8969-0e5181664b3a"
Sep 17 17:57:20.044501 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:20.044475    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:57:20.044893 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:20.044551    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98c4r" podUID="c53ddf27-1765-4785-b122-1645ccc8d094"
Sep 17 17:57:20.044893 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:20.044558    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:20.044893 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:20.044674    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shvtg" podUID="41ab02fa-694e-4b41-8969-0e5181664b3a"
Sep 17 17:57:20.044893 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:20.044735    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:57:20.044893 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:20.044834    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsz4t" podUID="11b6807f-cb26-4117-b06d-eb974e5640ae"
Sep 17 17:57:21.141238 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:21.140954    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm8hx_913ce1e9-9fb6-4049-a06d-9ebc69d3e528/ovn-acl-logging/0.log"
Sep 17 17:57:21.141807 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:21.141545    2581 generic.go:356] "Generic (PLEG): container finished" podID="913ce1e9-9fb6-4049-a06d-9ebc69d3e528" containerID="5b3ff4e650509c51ad91e2058cbba9936540110ed64dc6b618fc69581b5a8641" exitCode=1
Sep 17 17:57:21.141807 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:21.141614    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx" event={"ID":"913ce1e9-9fb6-4049-a06d-9ebc69d3e528","Type":"ContainerStarted","Data":"85f450f9dcf3ce5a3144917868ec33455570018bdf950701193bc305f741f3e1"}
Sep 17 17:57:21.141807 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:21.141707    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx" event={"ID":"913ce1e9-9fb6-4049-a06d-9ebc69d3e528","Type":"ContainerStarted","Data":"dae7db7daad259bf8caff8427881003cb1a9bcc2b75a7a1694856a55d8d0b511"}
Sep 17 17:57:21.141807 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:21.141749    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx" event={"ID":"913ce1e9-9fb6-4049-a06d-9ebc69d3e528","Type":"ContainerStarted","Data":"c66c2d6706ad845225aafec223aabf8f19b534ed7a82ede676aaca1d0544b4aa"}
Sep 17 17:57:21.141807 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:21.141777    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx" event={"ID":"913ce1e9-9fb6-4049-a06d-9ebc69d3e528","Type":"ContainerStarted","Data":"b7d5961e528a2c67f86d44a7863d8c18fc3bcd283e3bc3f3fdf47a8eabe1d7fc"}
Sep 17 17:57:21.141986 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:21.141814    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx" event={"ID":"913ce1e9-9fb6-4049-a06d-9ebc69d3e528","Type":"ContainerDied","Data":"5b3ff4e650509c51ad91e2058cbba9936540110ed64dc6b618fc69581b5a8641"}
Sep 17 17:57:21.141986 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:21.141855    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx" event={"ID":"913ce1e9-9fb6-4049-a06d-9ebc69d3e528","Type":"ContainerStarted","Data":"8e72b82b24c51a240e74b3c1059a0acc050a5b9fd023332412db00dc5f863234"}
Sep 17 17:57:21.142888 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:21.142867    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k7mgr" event={"ID":"383b592e-5a46-4299-9522-ac835c0adc61","Type":"ContainerStarted","Data":"912f4aa493907590abac494ea4c640099fa3e96ea69964cd9b5a413aaa15340c"}
Sep 17 17:57:21.144462 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:21.144439    2581 generic.go:356] "Generic (PLEG): container finished" podID="38076ecb-73ae-4c1c-8359-b9e2b7435a7b" containerID="23635547b9a6ed322ac052761662557b12194c4ab163aee68b3c7441d9c16ebd" exitCode=0
Sep 17 17:57:21.144602 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:21.144551    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8qdrq" event={"ID":"38076ecb-73ae-4c1c-8359-b9e2b7435a7b","Type":"ContainerDied","Data":"23635547b9a6ed322ac052761662557b12194c4ab163aee68b3c7441d9c16ebd"}
Sep 17 17:57:21.146606 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:21.146587    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7czlw" event={"ID":"ddb24cd0-f80d-426b-9910-71b2b1a4d578","Type":"ContainerStarted","Data":"148d676edd3ddf24ed8473853f9e97994e599f78fffd8eed8a3df55e3fb2199e"}
Sep 17 17:57:21.148674 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:21.148650    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mvn46" event={"ID":"7ba4edf3-c791-4134-9ea3-449a5c3b437c","Type":"ContainerStarted","Data":"e9c4a48d1142a087d48a5be96c9aa1ba10bc5383f1e5c47c01dd80c00ac430be"}
Sep 17 17:57:21.150376 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:21.150352    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xhbcn" event={"ID":"bd3ffd9f-338b-48ca-97c2-4292e8716182","Type":"ContainerStarted","Data":"fdad4933adfdba43d0bcda0578f9400cec2138472ce67bb2f913536980640438"}
Sep 17 17:57:21.154365 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:21.154345    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sf7g4" event={"ID":"ca53e15c-51d9-40c8-9d87-80e217ce1815","Type":"ContainerStarted","Data":"6bfe03ee24acc21a0141681cfe8a872be1e22b2d248ceab32394e2d351c8778e"}
Sep 17 17:57:21.156839 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:21.156816    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2wq8v" event={"ID":"6345bbfb-99aa-42ba-b5c9-d7c46ef2392f","Type":"ContainerStarted","Data":"e19f15b3931634992b0ba8f799d9409c276df530e5df68f2855171cfd9fc2735"}
Sep 17 17:57:21.163445 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:21.163409    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-209.ec2.internal" podStartSLOduration=21.163397808 podStartE2EDuration="21.163397808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:57:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:05.123501826 +0000 UTC m=+7.223965645" watchObservedRunningTime="2026-09-17 17:57:21.163397808 +0000 UTC m=+23.263861625"
Sep 17 17:57:21.163586 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:21.163566    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-k7mgr" podStartSLOduration=3.7703897570000002 podStartE2EDuration="21.163560762s" totalImagesPullingTime="17.393171005s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:21.163071491 +0000 UTC m=+23.263535310" watchObservedRunningTime="2026-09-17 17:57:21.163560762 +0000 UTC m=+23.264024614"
Sep 17 17:57:21.185304 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:21.185267    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xhbcn" podStartSLOduration=3.7662901 podStartE2EDuration="21.18525319s" totalImagesPullingTime="17.41896309s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:21.184997023 +0000 UTC m=+23.285460834" watchObservedRunningTime="2026-09-17 17:57:21.18525319 +0000 UTC m=+23.285717011"
Sep 17 17:57:21.201373 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:21.201333    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2wq8v" podStartSLOduration=3.872963628 podStartE2EDuration="21.201318824s" totalImagesPullingTime="17.328355196s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:21.200765527 +0000 UTC m=+23.301229348" watchObservedRunningTime="2026-09-17 17:57:21.201318824 +0000 UTC m=+23.301782646"
Sep 17 17:57:21.258942 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:21.258874    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sf7g4" podStartSLOduration=3.921168994 podStartE2EDuration="21.258854427s" totalImagesPullingTime="17.337685433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:21.221129187 +0000 UTC m=+23.321593005" watchObservedRunningTime="2026-09-17 17:57:21.258854427 +0000 UTC m=+23.359318249"
Sep 17 17:57:21.306853 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:21.306777    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mvn46" podStartSLOduration=3.917537488 podStartE2EDuration="21.306763462s" totalImagesPullingTime="17.389225974s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:21.259986898 +0000 UTC m=+23.360450718" watchObservedRunningTime="2026-09-17 17:57:21.306763462 +0000 UTC m=+23.407227280"
Sep 17 17:57:22.044046 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:22.044019    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:57:22.044046 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:22.044045    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:22.044233 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:22.044024    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:57:22.044233 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:22.044135    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98c4r" podUID="c53ddf27-1765-4785-b122-1645ccc8d094"
Sep 17 17:57:22.044338 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:22.044248    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsz4t" podUID="11b6807f-cb26-4117-b06d-eb974e5640ae"
Sep 17 17:57:22.044338 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:22.044314    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shvtg" podUID="41ab02fa-694e-4b41-8969-0e5181664b3a"
Sep 17 17:57:22.052704 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:22.052686    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 17:57:22.160559 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:22.160478    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4qkk8" event={"ID":"b4f2a63c-9017-431f-bec9-47521af4d1bd","Type":"ContainerStarted","Data":"5ec601c3100dea289f6f8b21ca14a6a21ebb4153419b57deb2109adfa495467c"}
Sep 17 17:57:22.162807 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:22.162779    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7czlw" event={"ID":"ddb24cd0-f80d-426b-9910-71b2b1a4d578","Type":"ContainerStarted","Data":"53187454c228fbc74f4452795d82a024f376167578c2b7cee186a0ff6849a2fb"}
Sep 17 17:57:22.179571 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:22.179530    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4qkk8" podStartSLOduration=4.778135278 podStartE2EDuration="22.179515882s" totalImagesPullingTime="17.401380604s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:22.179226467 +0000 UTC m=+24.279690282" watchObservedRunningTime="2026-09-17 17:57:22.179515882 +0000 UTC m=+24.279979700"
Sep 17 17:57:22.978594 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:22.978440    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T17:57:22.052701255Z","UUID":"dfe092ff-75e3-43e2-8a68-c029bb139f91","Handler":null,"Name":"","Endpoint":""}
Sep 17 17:57:22.980501 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:22.980478    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 17:57:22.980635 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:22.980514    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 17:57:23.168025 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:23.168000    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7czlw" event={"ID":"ddb24cd0-f80d-426b-9910-71b2b1a4d578","Type":"ContainerStarted","Data":"87266367b94b59e149c03ac9b30fe56aa74ab3fbd6ac106f9e16cc5f8b97a79c"}
Sep 17 17:57:23.848392 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:23.848359    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sf7g4"
Sep 17 17:57:23.849027 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:23.849005    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sf7g4"
Sep 17 17:57:23.865494 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:23.865452    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7czlw" podStartSLOduration=4.235695961 podStartE2EDuration="23.865439299s" totalImagesPullingTime="19.629743338s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:57:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:23.203717234 +0000 UTC m=+25.304181054" watchObservedRunningTime="2026-09-17 17:57:23.865439299 +0000 UTC m=+25.965903121"
Sep 17 17:57:24.047747 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:24.047687    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:24.047905 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:24.047787    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:57:24.047905 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:24.047802    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shvtg" podUID="41ab02fa-694e-4b41-8969-0e5181664b3a"
Sep 17 17:57:24.047905 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:24.047818    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:57:24.048054 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:24.047917    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98c4r" podUID="c53ddf27-1765-4785-b122-1645ccc8d094"
Sep 17 17:57:24.048054 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:24.048008    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsz4t" podUID="11b6807f-cb26-4117-b06d-eb974e5640ae"
Sep 17 17:57:24.173775 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:24.173748    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm8hx_913ce1e9-9fb6-4049-a06d-9ebc69d3e528/ovn-acl-logging/0.log"
Sep 17 17:57:24.174221 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:24.174189    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx" event={"ID":"913ce1e9-9fb6-4049-a06d-9ebc69d3e528","Type":"ContainerStarted","Data":"43f9546899e0b92cfee09c8aa4046ea33d1b3393a5def79b5efe8c1926ca87c0"}
Sep 17 17:57:26.043981 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:26.043793    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:57:26.044663 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:26.043822    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:57:26.044663 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:26.044060    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsz4t" podUID="11b6807f-cb26-4117-b06d-eb974e5640ae"
Sep 17 17:57:26.044663 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:26.044111    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98c4r" podUID="c53ddf27-1765-4785-b122-1645ccc8d094"
Sep 17 17:57:26.044663 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:26.043851    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:26.044663 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:26.044249    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shvtg" podUID="41ab02fa-694e-4b41-8969-0e5181664b3a"
Sep 17 17:57:26.181743 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:26.181712    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm8hx_913ce1e9-9fb6-4049-a06d-9ebc69d3e528/ovn-acl-logging/0.log"
Sep 17 17:57:26.182117 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:26.182093    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx" event={"ID":"913ce1e9-9fb6-4049-a06d-9ebc69d3e528","Type":"ContainerStarted","Data":"5be33124cba1128d0ef3109ffa82c3f772df6620baa383da5dbfffd1022a4ad0"}
Sep 17 17:57:26.182378 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:26.182354    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:26.182539 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:26.182504    2581 scope.go:117] "RemoveContainer" containerID="5b3ff4e650509c51ad91e2058cbba9936540110ed64dc6b618fc69581b5a8641"
Sep 17 17:57:26.184071 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:26.184052    2581 generic.go:356] "Generic (PLEG): container finished" podID="38076ecb-73ae-4c1c-8359-b9e2b7435a7b" containerID="4c20c8c5706418bd69a7eb4038f55c293b2b8c16065524e17b63b9e9be97a299" exitCode=0
Sep 17 17:57:26.184177 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:26.184087    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8qdrq" event={"ID":"38076ecb-73ae-4c1c-8359-b9e2b7435a7b","Type":"ContainerDied","Data":"4c20c8c5706418bd69a7eb4038f55c293b2b8c16065524e17b63b9e9be97a299"}
Sep 17 17:57:26.198686 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:26.198669    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:27.190135 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:27.189975    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm8hx_913ce1e9-9fb6-4049-a06d-9ebc69d3e528/ovn-acl-logging/0.log"
Sep 17 17:57:27.190542 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:27.190518    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx" event={"ID":"913ce1e9-9fb6-4049-a06d-9ebc69d3e528","Type":"ContainerStarted","Data":"135a033f7a0c6096774c6e240aa5ceb90b83662f73cbbef337d0b1c659a36c1b"}
Sep 17 17:57:27.190770 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:27.190751    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:27.190832 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:27.190822    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:27.206542 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:27.206521    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:27.225470 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:27.225428    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx" podStartSLOduration=9.834274868 podStartE2EDuration="27.225415338s" totalImagesPullingTime="17.39114047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:27.224797289 +0000 UTC m=+29.325261108" watchObservedRunningTime="2026-09-17 17:57:27.225415338 +0000 UTC m=+29.325879195"
Sep 17 17:57:27.659347 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:27.659322    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-98c4r"]
Sep 17 17:57:27.659520 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:27.659432    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:57:27.659581 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:27.659531    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98c4r" podUID="c53ddf27-1765-4785-b122-1645ccc8d094"
Sep 17 17:57:27.664021 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:27.663987    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dsz4t"]
Sep 17 17:57:27.664170 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:27.664096    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:57:27.664240 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:27.664219    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsz4t" podUID="11b6807f-cb26-4117-b06d-eb974e5640ae"
Sep 17 17:57:27.664661 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:27.664637    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-shvtg"]
Sep 17 17:57:27.664791 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:27.664706    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:27.664791 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:27.664773    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shvtg" podUID="41ab02fa-694e-4b41-8969-0e5181664b3a"
Sep 17 17:57:28.197318 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:28.197285    2581 generic.go:356] "Generic (PLEG): container finished" podID="38076ecb-73ae-4c1c-8359-b9e2b7435a7b" containerID="35e980530c0bb85c932e6c808a80ab2fb0a51728179cc1382ab180417f0d0da4" exitCode=0
Sep 17 17:57:28.197679 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:28.197365    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8qdrq" event={"ID":"38076ecb-73ae-4c1c-8359-b9e2b7435a7b","Type":"ContainerDied","Data":"35e980530c0bb85c932e6c808a80ab2fb0a51728179cc1382ab180417f0d0da4"}
Sep 17 17:57:29.043325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:29.043295    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:29.043476 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:29.043401    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shvtg" podUID="41ab02fa-694e-4b41-8969-0e5181664b3a"
Sep 17 17:57:29.043476 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:29.043449    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:57:29.043552 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:29.043538    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsz4t" podUID="11b6807f-cb26-4117-b06d-eb974e5640ae"
Sep 17 17:57:30.044011 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:30.043976    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:57:30.044449 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:30.044058    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98c4r" podUID="c53ddf27-1765-4785-b122-1645ccc8d094"
Sep 17 17:57:30.205009 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:30.204974    2581 generic.go:356] "Generic (PLEG): container finished" podID="38076ecb-73ae-4c1c-8359-b9e2b7435a7b" containerID="0ee4003f7de8cc7b8f9fe313b44807a9e9933d56ed27d7354e8d8918bb46c268" exitCode=0
Sep 17 17:57:30.205217 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:30.205057    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8qdrq" event={"ID":"38076ecb-73ae-4c1c-8359-b9e2b7435a7b","Type":"ContainerDied","Data":"0ee4003f7de8cc7b8f9fe313b44807a9e9933d56ed27d7354e8d8918bb46c268"}
Sep 17 17:57:31.043992 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:31.043961    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:57:31.044219 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:31.043963    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:31.044219 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:31.044087    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsz4t" podUID="11b6807f-cb26-4117-b06d-eb974e5640ae"
Sep 17 17:57:31.044219 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:31.044124    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shvtg" podUID="41ab02fa-694e-4b41-8969-0e5181664b3a"
Sep 17 17:57:31.387418 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:31.387386    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sf7g4"
Sep 17 17:57:31.387604 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:31.387518    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 17:57:31.388224 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:31.388204    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sf7g4"
Sep 17 17:57:32.043730 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:32.043697    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:57:32.043919 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:32.043811    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98c4r" podUID="c53ddf27-1765-4785-b122-1645ccc8d094"
Sep 17 17:57:33.044118 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:33.043877    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:33.044538 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:33.043943    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:57:33.044538 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:33.044239    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shvtg" podUID="41ab02fa-694e-4b41-8969-0e5181664b3a"
Sep 17 17:57:33.044538 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:33.044306    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsz4t" podUID="11b6807f-cb26-4117-b06d-eb974e5640ae"
Sep 17 17:57:33.648965 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:33.648923    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11b6807f-cb26-4117-b06d-eb974e5640ae-metrics-certs\") pod \"network-metrics-daemon-dsz4t\" (UID: \"11b6807f-cb26-4117-b06d-eb974e5640ae\") " pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:57:33.649141 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:33.649057    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:33.649141 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:33.649130    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11b6807f-cb26-4117-b06d-eb974e5640ae-metrics-certs podName:11b6807f-cb26-4117-b06d-eb974e5640ae nodeName:}" failed. No retries permitted until 2026-09-17 17:58:05.649108509 +0000 UTC m=+67.749572307 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11b6807f-cb26-4117-b06d-eb974e5640ae-metrics-certs") pod "network-metrics-daemon-dsz4t" (UID: "11b6807f-cb26-4117-b06d-eb974e5640ae") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:33.750106 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:33.750069    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41ab02fa-694e-4b41-8969-0e5181664b3a-original-pull-secret\") pod \"global-pull-secret-syncer-shvtg\" (UID: \"41ab02fa-694e-4b41-8969-0e5181664b3a\") " pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:33.750278 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:33.750120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2hn2\" (UniqueName: \"kubernetes.io/projected/c53ddf27-1765-4785-b122-1645ccc8d094-kube-api-access-p2hn2\") pod \"network-check-target-98c4r\" (UID: \"c53ddf27-1765-4785-b122-1645ccc8d094\") " pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:57:33.750278 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:33.750273    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:57:33.750347 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:33.750287    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:57:33.750347 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:33.750296    2581 projected.go:196] Error preparing data for projected volume kube-api-access-p2hn2 for pod openshift-network-diagnostics/network-check-target-98c4r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:33.750347 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:33.750338    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c53ddf27-1765-4785-b122-1645ccc8d094-kube-api-access-p2hn2 podName:c53ddf27-1765-4785-b122-1645ccc8d094 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:05.750323602 +0000 UTC m=+67.850787412 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-p2hn2" (UniqueName: "kubernetes.io/projected/c53ddf27-1765-4785-b122-1645ccc8d094-kube-api-access-p2hn2") pod "network-check-target-98c4r" (UID: "c53ddf27-1765-4785-b122-1645ccc8d094") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:33.750771 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:33.750692    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:33.750771 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:33.750739    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41ab02fa-694e-4b41-8969-0e5181664b3a-original-pull-secret podName:41ab02fa-694e-4b41-8969-0e5181664b3a nodeName:}" failed. No retries permitted until 2026-09-17 17:58:05.750724131 +0000 UTC m=+67.851187942 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41ab02fa-694e-4b41-8969-0e5181664b3a-original-pull-secret") pod "global-pull-secret-syncer-shvtg" (UID: "41ab02fa-694e-4b41-8969-0e5181664b3a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:34.044173 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.044127    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:57:34.044553 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:34.044246    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98c4r" podUID="c53ddf27-1765-4785-b122-1645ccc8d094"
Sep 17 17:57:34.434739 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.434713    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-209.ec2.internal" event="NodeReady"
Sep 17 17:57:34.434876 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.434858    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 17:57:34.494521 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.494485    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jhhsj"]
Sep 17 17:57:34.496344 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.496326    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-p7h8g"]
Sep 17 17:57:34.496455 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.496440    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jhhsj"
Sep 17 17:57:34.498054 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.498034    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-p7h8g"
Sep 17 17:57:34.498546 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.498507    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 17:57:34.498742 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.498726    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 17:57:34.499080 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.499065    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zr8ck\""
Sep 17 17:57:34.499994 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.499974    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 17:57:34.499994 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.499992    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 17:57:34.500141 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.500102    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 17:57:34.500444 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.500426    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5ghb8\""
Sep 17 17:57:34.513649 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.513625    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jhhsj"]
Sep 17 17:57:34.515591 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.515034    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-p7h8g"]
Sep 17 17:57:34.655784 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.655755    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl58c\" (UniqueName: \"kubernetes.io/projected/a1d64f49-582d-4389-b190-b84fae699d42-kube-api-access-hl58c\") pod \"dns-default-jhhsj\" (UID: \"a1d64f49-582d-4389-b190-b84fae699d42\") " pod="openshift-dns/dns-default-jhhsj"
Sep 17 17:57:34.655916 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.655805    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a1d64f49-582d-4389-b190-b84fae699d42-config-volume\") pod \"dns-default-jhhsj\" (UID: \"a1d64f49-582d-4389-b190-b84fae699d42\") " pod="openshift-dns/dns-default-jhhsj"
Sep 17 17:57:34.655916 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.655899    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-989lp\" (UniqueName: \"kubernetes.io/projected/47da4adc-a0aa-4908-91b7-d3baae40714d-kube-api-access-989lp\") pod \"ingress-canary-p7h8g\" (UID: \"47da4adc-a0aa-4908-91b7-d3baae40714d\") " pod="openshift-ingress-canary/ingress-canary-p7h8g"
Sep 17 17:57:34.656023 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.655931    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1d64f49-582d-4389-b190-b84fae699d42-tmp-dir\") pod \"dns-default-jhhsj\" (UID: \"a1d64f49-582d-4389-b190-b84fae699d42\") " pod="openshift-dns/dns-default-jhhsj"
Sep 17 17:57:34.656023 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.655954    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1d64f49-582d-4389-b190-b84fae699d42-metrics-tls\") pod \"dns-default-jhhsj\" (UID: \"a1d64f49-582d-4389-b190-b84fae699d42\") " pod="openshift-dns/dns-default-jhhsj"
Sep 17 17:57:34.656023 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.655971    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47da4adc-a0aa-4908-91b7-d3baae40714d-cert\") pod \"ingress-canary-p7h8g\" (UID: \"47da4adc-a0aa-4908-91b7-d3baae40714d\") " pod="openshift-ingress-canary/ingress-canary-p7h8g"
Sep 17 17:57:34.756873 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.756846    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a1d64f49-582d-4389-b190-b84fae699d42-config-volume\") pod \"dns-default-jhhsj\" (UID: \"a1d64f49-582d-4389-b190-b84fae699d42\") " pod="openshift-dns/dns-default-jhhsj"
Sep 17 17:57:34.756997 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.756888    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-989lp\" (UniqueName: \"kubernetes.io/projected/47da4adc-a0aa-4908-91b7-d3baae40714d-kube-api-access-989lp\") pod \"ingress-canary-p7h8g\" (UID: \"47da4adc-a0aa-4908-91b7-d3baae40714d\") " pod="openshift-ingress-canary/ingress-canary-p7h8g"
Sep 17 17:57:34.756997 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.756917    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1d64f49-582d-4389-b190-b84fae699d42-tmp-dir\") pod \"dns-default-jhhsj\" (UID: \"a1d64f49-582d-4389-b190-b84fae699d42\") " pod="openshift-dns/dns-default-jhhsj"
Sep 17 17:57:34.756997 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.756948    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1d64f49-582d-4389-b190-b84fae699d42-metrics-tls\") pod \"dns-default-jhhsj\" (UID: \"a1d64f49-582d-4389-b190-b84fae699d42\") " pod="openshift-dns/dns-default-jhhsj"
Sep 17 17:57:34.756997 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.756969    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47da4adc-a0aa-4908-91b7-d3baae40714d-cert\") pod \"ingress-canary-p7h8g\" (UID: \"47da4adc-a0aa-4908-91b7-d3baae40714d\") " pod="openshift-ingress-canary/ingress-canary-p7h8g"
Sep 17 17:57:34.757229 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.757018    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl58c\" (UniqueName: \"kubernetes.io/projected/a1d64f49-582d-4389-b190-b84fae699d42-kube-api-access-hl58c\") pod \"dns-default-jhhsj\" (UID: \"a1d64f49-582d-4389-b190-b84fae699d42\") " pod="openshift-dns/dns-default-jhhsj"
Sep 17 17:57:34.757229 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:34.757098    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:34.757229 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:34.757145    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1d64f49-582d-4389-b190-b84fae699d42-metrics-tls podName:a1d64f49-582d-4389-b190-b84fae699d42 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:35.257131074 +0000 UTC m=+37.357594871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a1d64f49-582d-4389-b190-b84fae699d42-metrics-tls") pod "dns-default-jhhsj" (UID: "a1d64f49-582d-4389-b190-b84fae699d42") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:34.757378 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.757346    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1d64f49-582d-4389-b190-b84fae699d42-tmp-dir\") pod \"dns-default-jhhsj\" (UID: \"a1d64f49-582d-4389-b190-b84fae699d42\") " pod="openshift-dns/dns-default-jhhsj"
Sep 17 17:57:34.757510 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.757493    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a1d64f49-582d-4389-b190-b84fae699d42-config-volume\") pod \"dns-default-jhhsj\" (UID: \"a1d64f49-582d-4389-b190-b84fae699d42\") " pod="openshift-dns/dns-default-jhhsj"
Sep 17 17:57:34.760980 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.760962    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47da4adc-a0aa-4908-91b7-d3baae40714d-cert\") pod \"ingress-canary-p7h8g\" (UID: \"47da4adc-a0aa-4908-91b7-d3baae40714d\") " pod="openshift-ingress-canary/ingress-canary-p7h8g"
Sep 17 17:57:34.768234 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.768214    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl58c\" (UniqueName: \"kubernetes.io/projected/a1d64f49-582d-4389-b190-b84fae699d42-kube-api-access-hl58c\") pod \"dns-default-jhhsj\" (UID: \"a1d64f49-582d-4389-b190-b84fae699d42\") " pod="openshift-dns/dns-default-jhhsj"
Sep 17 17:57:34.768640 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.768624    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-989lp\" (UniqueName: \"kubernetes.io/projected/47da4adc-a0aa-4908-91b7-d3baae40714d-kube-api-access-989lp\") pod \"ingress-canary-p7h8g\" (UID: \"47da4adc-a0aa-4908-91b7-d3baae40714d\") " pod="openshift-ingress-canary/ingress-canary-p7h8g"
Sep 17 17:57:34.811366 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:34.811336    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-p7h8g"
Sep 17 17:57:35.043723 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:35.043660    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:57:35.043837 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:35.043670    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:57:35.049348 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:35.049324    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:57:35.049348 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:35.049340    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s7697\""
Sep 17 17:57:35.050067 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:35.049365    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:57:35.091675 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:35.091649    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-p7h8g"]
Sep 17 17:57:35.095343 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:57:35.095324    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47da4adc_a0aa_4908_91b7_d3baae40714d.slice/crio-8c0c3602bb730911d63766fbca1703e55751fac46c4d7b42cc46bc5c2f6cde84 WatchSource:0}: Error finding container 8c0c3602bb730911d63766fbca1703e55751fac46c4d7b42cc46bc5c2f6cde84: Status 404 returned error can't find the container with id 8c0c3602bb730911d63766fbca1703e55751fac46c4d7b42cc46bc5c2f6cde84
Sep 17 17:57:35.216673 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:35.216637    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-p7h8g" event={"ID":"47da4adc-a0aa-4908-91b7-d3baae40714d","Type":"ContainerStarted","Data":"8c0c3602bb730911d63766fbca1703e55751fac46c4d7b42cc46bc5c2f6cde84"}
Sep 17 17:57:35.219412 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:35.219390    2581 generic.go:356] "Generic (PLEG): container finished" podID="38076ecb-73ae-4c1c-8359-b9e2b7435a7b" containerID="a6a1a791fd6023427e164ed43f71d0ad525f13188c54fceb9e16197ec7e83b9e" exitCode=0
Sep 17 17:57:35.219493 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:35.219427    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8qdrq" event={"ID":"38076ecb-73ae-4c1c-8359-b9e2b7435a7b","Type":"ContainerDied","Data":"a6a1a791fd6023427e164ed43f71d0ad525f13188c54fceb9e16197ec7e83b9e"}
Sep 17 17:57:35.260095 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:35.260071    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1d64f49-582d-4389-b190-b84fae699d42-metrics-tls\") pod \"dns-default-jhhsj\" (UID: \"a1d64f49-582d-4389-b190-b84fae699d42\") " pod="openshift-dns/dns-default-jhhsj"
Sep 17 17:57:35.260241 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:35.260223    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:35.260304 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:35.260286    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1d64f49-582d-4389-b190-b84fae699d42-metrics-tls podName:a1d64f49-582d-4389-b190-b84fae699d42 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:36.260270244 +0000 UTC m=+38.360734042 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a1d64f49-582d-4389-b190-b84fae699d42-metrics-tls") pod "dns-default-jhhsj" (UID: "a1d64f49-582d-4389-b190-b84fae699d42") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:36.043899 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:36.043709    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:57:36.046669 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:36.046641    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:57:36.047021 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:36.046998    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6psft\""
Sep 17 17:57:36.047713 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:36.047690    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:57:36.226584 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:36.226537    2581 generic.go:356] "Generic (PLEG): container finished" podID="38076ecb-73ae-4c1c-8359-b9e2b7435a7b" containerID="262cb48e5b127f5f5089482fe1ab2e8ed71395cd25a1a8d155b2abd15fc9d966" exitCode=0
Sep 17 17:57:36.227030 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:36.226602    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8qdrq" event={"ID":"38076ecb-73ae-4c1c-8359-b9e2b7435a7b","Type":"ContainerDied","Data":"262cb48e5b127f5f5089482fe1ab2e8ed71395cd25a1a8d155b2abd15fc9d966"}
Sep 17 17:57:36.269207 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:36.268822    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1d64f49-582d-4389-b190-b84fae699d42-metrics-tls\") pod \"dns-default-jhhsj\" (UID: \"a1d64f49-582d-4389-b190-b84fae699d42\") " pod="openshift-dns/dns-default-jhhsj"
Sep 17 17:57:36.269207 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:36.268960    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:36.269207 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:36.269032    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1d64f49-582d-4389-b190-b84fae699d42-metrics-tls podName:a1d64f49-582d-4389-b190-b84fae699d42 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:38.269013498 +0000 UTC m=+40.369477318 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a1d64f49-582d-4389-b190-b84fae699d42-metrics-tls") pod "dns-default-jhhsj" (UID: "a1d64f49-582d-4389-b190-b84fae699d42") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:37.231073 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:37.230984    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-p7h8g" event={"ID":"47da4adc-a0aa-4908-91b7-d3baae40714d","Type":"ContainerStarted","Data":"db10007d81f50a4185d70467af121147a98bff4b677fa0caa5388c108dadad2a"}
Sep 17 17:57:37.234564 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:37.234537    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8qdrq" event={"ID":"38076ecb-73ae-4c1c-8359-b9e2b7435a7b","Type":"ContainerStarted","Data":"29ae04a36c06a10e2d90930a79c6846bd55e9c451a6f58eb57dffeccbba73306"}
Sep 17 17:57:37.260269 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:37.260220    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-p7h8g" podStartSLOduration=1.2776281950000001 podStartE2EDuration="3.260205336s" totalImagesPullingTime="1.982577141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:37.257187731 +0000 UTC m=+39.357651548" watchObservedRunningTime="2026-09-17 17:57:37.260205336 +0000 UTC m=+39.360669155"
Sep 17 17:57:37.298553 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:37.298509    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8qdrq" podStartSLOduration=9.316077704 podStartE2EDuration="37.298492982s" totalImagesPullingTime="27.982415278s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:57:00 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:37.29802898 +0000 UTC m=+39.398492800" watchObservedRunningTime="2026-09-17 17:57:37.298492982 +0000 UTC m=+39.398956795"
Sep 17 17:57:38.282760 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:38.282723    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1d64f49-582d-4389-b190-b84fae699d42-metrics-tls\") pod \"dns-default-jhhsj\" (UID: \"a1d64f49-582d-4389-b190-b84fae699d42\") " pod="openshift-dns/dns-default-jhhsj"
Sep 17 17:57:38.283115 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:38.282872    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:38.283115 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:38.282934    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1d64f49-582d-4389-b190-b84fae699d42-metrics-tls podName:a1d64f49-582d-4389-b190-b84fae699d42 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:42.282920007 +0000 UTC m=+44.383383808 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a1d64f49-582d-4389-b190-b84fae699d42-metrics-tls") pod "dns-default-jhhsj" (UID: "a1d64f49-582d-4389-b190-b84fae699d42") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:42.314602 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:42.314570    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1d64f49-582d-4389-b190-b84fae699d42-metrics-tls\") pod \"dns-default-jhhsj\" (UID: \"a1d64f49-582d-4389-b190-b84fae699d42\") " pod="openshift-dns/dns-default-jhhsj"
Sep 17 17:57:42.315048 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:42.314719    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:42.315048 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:42.314779    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1d64f49-582d-4389-b190-b84fae699d42-metrics-tls podName:a1d64f49-582d-4389-b190-b84fae699d42 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:50.314765137 +0000 UTC m=+52.415228939 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a1d64f49-582d-4389-b190-b84fae699d42-metrics-tls") pod "dns-default-jhhsj" (UID: "a1d64f49-582d-4389-b190-b84fae699d42") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:50.369047 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:50.369015    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1d64f49-582d-4389-b190-b84fae699d42-metrics-tls\") pod \"dns-default-jhhsj\" (UID: \"a1d64f49-582d-4389-b190-b84fae699d42\") " pod="openshift-dns/dns-default-jhhsj"
Sep 17 17:57:50.369418 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:50.369130    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:50.369418 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:50.369197    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1d64f49-582d-4389-b190-b84fae699d42-metrics-tls podName:a1d64f49-582d-4389-b190-b84fae699d42 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:06.369184718 +0000 UTC m=+68.469648515 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a1d64f49-582d-4389-b190-b84fae699d42-metrics-tls") pod "dns-default-jhhsj" (UID: "a1d64f49-582d-4389-b190-b84fae699d42") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:56.926020 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:56.925990    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-ztc5j"]
Sep 17 17:57:56.956236 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:56.956211    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-ztc5j"]
Sep 17 17:57:56.956380 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:56.956315    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-ztc5j"
Sep 17 17:57:56.958517 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:56.958493    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 17:57:56.958621 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:56.958493    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rbnt5\""
Sep 17 17:57:56.959250 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:56.959235    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 17:57:56.959475 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:56.959456    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 17:57:56.960391 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:56.960377    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 17:57:56.965621 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:56.965605    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 17:57:57.009358 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.009325    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1080ecc5-cca9-47db-b949-05bdd3777351-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-ztc5j\" (UID: \"1080ecc5-cca9-47db-b949-05bdd3777351\") " pod="openshift-insights/insights-operator-6f9c644dd9-ztc5j"
Sep 17 17:57:57.009527 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.009375    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pqgj\" (UniqueName: \"kubernetes.io/projected/1080ecc5-cca9-47db-b949-05bdd3777351-kube-api-access-2pqgj\") pod \"insights-operator-6f9c644dd9-ztc5j\" (UID: \"1080ecc5-cca9-47db-b949-05bdd3777351\") " pod="openshift-insights/insights-operator-6f9c644dd9-ztc5j"
Sep 17 17:57:57.009527 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.009410    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1080ecc5-cca9-47db-b949-05bdd3777351-tmp\") pod \"insights-operator-6f9c644dd9-ztc5j\" (UID: \"1080ecc5-cca9-47db-b949-05bdd3777351\") " pod="openshift-insights/insights-operator-6f9c644dd9-ztc5j"
Sep 17 17:57:57.009527 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.009439    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1080ecc5-cca9-47db-b949-05bdd3777351-snapshots\") pod \"insights-operator-6f9c644dd9-ztc5j\" (UID: \"1080ecc5-cca9-47db-b949-05bdd3777351\") " pod="openshift-insights/insights-operator-6f9c644dd9-ztc5j"
Sep 17 17:57:57.009527 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.009462    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1080ecc5-cca9-47db-b949-05bdd3777351-serving-cert\") pod \"insights-operator-6f9c644dd9-ztc5j\" (UID: \"1080ecc5-cca9-47db-b949-05bdd3777351\") " pod="openshift-insights/insights-operator-6f9c644dd9-ztc5j"
Sep 17 17:57:57.009704 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.009527    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1080ecc5-cca9-47db-b949-05bdd3777351-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-ztc5j\" (UID: \"1080ecc5-cca9-47db-b949-05bdd3777351\") " pod="openshift-insights/insights-operator-6f9c644dd9-ztc5j"
Sep 17 17:57:57.025093 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.025061    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-rnprf"]
Sep 17 17:57:57.045621 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.045596    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-cfm6h"]
Sep 17 17:57:57.045838 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.045810    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-rnprf"
Sep 17 17:57:57.051595 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.051574    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:57:57.051733 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.051577    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-k9ffw\""
Sep 17 17:57:57.058910 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.058889    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 17 17:57:57.071789 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.071758    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-cfm6h"
Sep 17 17:57:57.071789 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.071766    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wt46p"]
Sep 17 17:57:57.075931 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.075909    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 17:57:57.076061 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.076039    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:57:57.076115 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.076098    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 17:57:57.076177 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.076140    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-k2f6m\""
Sep 17 17:57:57.076467 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.076453    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 17:57:57.095648 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.095625    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-cfm6h"]
Sep 17 17:57:57.095732 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.095654    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-rnprf"]
Sep 17 17:57:57.095732 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.095672    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wt46p"]
Sep 17 17:57:57.095795 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.095737    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wt46p"
Sep 17 17:57:57.098452 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.098435    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:57:57.098521 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.098470    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 17:57:57.098603 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.098589    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4g6xq\""
Sep 17 17:57:57.098917 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.098901    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 17:57:57.110378 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.110361    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1080ecc5-cca9-47db-b949-05bdd3777351-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-ztc5j\" (UID: \"1080ecc5-cca9-47db-b949-05bdd3777351\") " pod="openshift-insights/insights-operator-6f9c644dd9-ztc5j"
Sep 17 17:57:57.110549 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.110384    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pqgj\" (UniqueName: \"kubernetes.io/projected/1080ecc5-cca9-47db-b949-05bdd3777351-kube-api-access-2pqgj\") pod \"insights-operator-6f9c644dd9-ztc5j\" (UID: \"1080ecc5-cca9-47db-b949-05bdd3777351\") " pod="openshift-insights/insights-operator-6f9c644dd9-ztc5j"
Sep 17 17:57:57.110549 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.110403    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1080ecc5-cca9-47db-b949-05bdd3777351-tmp\") pod \"insights-operator-6f9c644dd9-ztc5j\" (UID: \"1080ecc5-cca9-47db-b949-05bdd3777351\") " pod="openshift-insights/insights-operator-6f9c644dd9-ztc5j"
Sep 17 17:57:57.110549 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.110418    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1080ecc5-cca9-47db-b949-05bdd3777351-snapshots\") pod \"insights-operator-6f9c644dd9-ztc5j\" (UID: \"1080ecc5-cca9-47db-b949-05bdd3777351\") " pod="openshift-insights/insights-operator-6f9c644dd9-ztc5j"
Sep 17 17:57:57.110549 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.110433    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1080ecc5-cca9-47db-b949-05bdd3777351-serving-cert\") pod \"insights-operator-6f9c644dd9-ztc5j\" (UID: \"1080ecc5-cca9-47db-b949-05bdd3777351\") " pod="openshift-insights/insights-operator-6f9c644dd9-ztc5j"
Sep 17 17:57:57.110549 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.110451    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zxdd\" (UniqueName: \"kubernetes.io/projected/0cd09d92-625e-4b02-898e-0ab0fb807c58-kube-api-access-5zxdd\") pod \"cluster-samples-operator-685d7d68b5-wt46p\" (UID: \"0cd09d92-625e-4b02-898e-0ab0fb807c58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wt46p"
Sep 17 17:57:57.110549 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.110469    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsm48\" (UniqueName: \"kubernetes.io/projected/184b0189-e1d0-4bfd-b970-596373fd0745-kube-api-access-hsm48\") pod \"kube-storage-version-migrator-operator-55889f7b47-cfm6h\" (UID: \"184b0189-e1d0-4bfd-b970-596373fd0745\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-cfm6h"
Sep 17 17:57:57.110549 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.110500    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/184b0189-e1d0-4bfd-b970-596373fd0745-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-cfm6h\" (UID: \"184b0189-e1d0-4bfd-b970-596373fd0745\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-cfm6h"
Sep 17 17:57:57.110549 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.110524    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/184b0189-e1d0-4bfd-b970-596373fd0745-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-cfm6h\" (UID: \"184b0189-e1d0-4bfd-b970-596373fd0745\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-cfm6h"
Sep 17 17:57:57.110549 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.110554    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1080ecc5-cca9-47db-b949-05bdd3777351-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-ztc5j\" (UID: \"1080ecc5-cca9-47db-b949-05bdd3777351\") " pod="openshift-insights/insights-operator-6f9c644dd9-ztc5j"
Sep 17 17:57:57.110935 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.110595    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6j58\" (UniqueName: \"kubernetes.io/projected/36a45fc1-bf32-4f4b-b741-75ac26197973-kube-api-access-f6j58\") pod \"volume-data-source-validator-5c4db8bdf9-rnprf\" (UID: \"36a45fc1-bf32-4f4b-b741-75ac26197973\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-rnprf"
Sep 17 17:57:57.110935 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.110618    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0cd09d92-625e-4b02-898e-0ab0fb807c58-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-wt46p\" (UID: \"0cd09d92-625e-4b02-898e-0ab0fb807c58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wt46p"
Sep 17 17:57:57.110935 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.110788    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1080ecc5-cca9-47db-b949-05bdd3777351-tmp\") pod \"insights-operator-6f9c644dd9-ztc5j\" (UID: \"1080ecc5-cca9-47db-b949-05bdd3777351\") " pod="openshift-insights/insights-operator-6f9c644dd9-ztc5j"
Sep 17 17:57:57.111065 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.110973    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1080ecc5-cca9-47db-b949-05bdd3777351-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-ztc5j\" (UID: \"1080ecc5-cca9-47db-b949-05bdd3777351\") " pod="openshift-insights/insights-operator-6f9c644dd9-ztc5j"
Sep 17 17:57:57.111116 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.111076    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1080ecc5-cca9-47db-b949-05bdd3777351-snapshots\") pod \"insights-operator-6f9c644dd9-ztc5j\" (UID: \"1080ecc5-cca9-47db-b949-05bdd3777351\") " pod="openshift-insights/insights-operator-6f9c644dd9-ztc5j"
Sep 17 17:57:57.111331 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.111312    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1080ecc5-cca9-47db-b949-05bdd3777351-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-ztc5j\" (UID: \"1080ecc5-cca9-47db-b949-05bdd3777351\") " pod="openshift-insights/insights-operator-6f9c644dd9-ztc5j"
Sep 17 17:57:57.113933 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.113917    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1080ecc5-cca9-47db-b949-05bdd3777351-serving-cert\") pod \"insights-operator-6f9c644dd9-ztc5j\" (UID: \"1080ecc5-cca9-47db-b949-05bdd3777351\") " pod="openshift-insights/insights-operator-6f9c644dd9-ztc5j"
Sep 17 17:57:57.125183 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.125128    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pqgj\" (UniqueName: \"kubernetes.io/projected/1080ecc5-cca9-47db-b949-05bdd3777351-kube-api-access-2pqgj\") pod \"insights-operator-6f9c644dd9-ztc5j\" (UID: \"1080ecc5-cca9-47db-b949-05bdd3777351\") " pod="openshift-insights/insights-operator-6f9c644dd9-ztc5j"
Sep 17 17:57:57.126072 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.126052    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6787978d5c-jv2ww"]
Sep 17 17:57:57.152132 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.152114    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:57:57.155449 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.155430    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 17:57:57.155539 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.155453    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 17:57:57.155599 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.155550    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lg89h\""
Sep 17 17:57:57.157977 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.157954    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 17:57:57.162174 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.162131    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6787978d5c-jv2ww"]
Sep 17 17:57:57.163708 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.163688    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 17:57:57.211717 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.211640    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zxdd\" (UniqueName: \"kubernetes.io/projected/0cd09d92-625e-4b02-898e-0ab0fb807c58-kube-api-access-5zxdd\") pod \"cluster-samples-operator-685d7d68b5-wt46p\" (UID: \"0cd09d92-625e-4b02-898e-0ab0fb807c58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wt46p"
Sep 17 17:57:57.211717 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.211675    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsm48\" (UniqueName: \"kubernetes.io/projected/184b0189-e1d0-4bfd-b970-596373fd0745-kube-api-access-hsm48\") pod \"kube-storage-version-migrator-operator-55889f7b47-cfm6h\" (UID: \"184b0189-e1d0-4bfd-b970-596373fd0745\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-cfm6h"
Sep 17 17:57:57.211717 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.211702    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d396cd04-2841-4a84-8fd3-235da3979f1c-registry-certificates\") pod \"image-registry-6787978d5c-jv2ww\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") " pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:57:57.211958 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.211725    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-bound-sa-token\") pod \"image-registry-6787978d5c-jv2ww\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") " pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:57:57.211958 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.211776    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/184b0189-e1d0-4bfd-b970-596373fd0745-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-cfm6h\" (UID: \"184b0189-e1d0-4bfd-b970-596373fd0745\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-cfm6h"
Sep 17 17:57:57.211958 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.211810    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/184b0189-e1d0-4bfd-b970-596373fd0745-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-cfm6h\" (UID: \"184b0189-e1d0-4bfd-b970-596373fd0745\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-cfm6h"
Sep 17 17:57:57.211958 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.211860    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d396cd04-2841-4a84-8fd3-235da3979f1c-ca-trust-extracted\") pod \"image-registry-6787978d5c-jv2ww\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") " pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:57:57.211958 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.211887    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d396cd04-2841-4a84-8fd3-235da3979f1c-image-registry-private-configuration\") pod \"image-registry-6787978d5c-jv2ww\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") " pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:57:57.211958 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.211913    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-registry-tls\") pod \"image-registry-6787978d5c-jv2ww\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") " pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:57:57.211958 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.211938    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d396cd04-2841-4a84-8fd3-235da3979f1c-trusted-ca\") pod \"image-registry-6787978d5c-jv2ww\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") " pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:57:57.212300 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.212001    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7mcf\" (UniqueName: \"kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-kube-api-access-r7mcf\") pod \"image-registry-6787978d5c-jv2ww\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") " pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:57:57.212300 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.212042    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6j58\" (UniqueName: \"kubernetes.io/projected/36a45fc1-bf32-4f4b-b741-75ac26197973-kube-api-access-f6j58\") pod \"volume-data-source-validator-5c4db8bdf9-rnprf\" (UID: \"36a45fc1-bf32-4f4b-b741-75ac26197973\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-rnprf"
Sep 17 17:57:57.212300 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.212060    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0cd09d92-625e-4b02-898e-0ab0fb807c58-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-wt46p\" (UID: \"0cd09d92-625e-4b02-898e-0ab0fb807c58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wt46p"
Sep 17 17:57:57.212300 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.212075    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d396cd04-2841-4a84-8fd3-235da3979f1c-installation-pull-secrets\") pod \"image-registry-6787978d5c-jv2ww\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") " pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:57:57.212300 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:57.212177    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:57:57.212300 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:57.212248    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cd09d92-625e-4b02-898e-0ab0fb807c58-samples-operator-tls podName:0cd09d92-625e-4b02-898e-0ab0fb807c58 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:57.712233968 +0000 UTC m=+59.812697765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0cd09d92-625e-4b02-898e-0ab0fb807c58-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-wt46p" (UID: "0cd09d92-625e-4b02-898e-0ab0fb807c58") : secret "samples-operator-tls" not found
Sep 17 17:57:57.212506 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.212432    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/184b0189-e1d0-4bfd-b970-596373fd0745-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-cfm6h\" (UID: \"184b0189-e1d0-4bfd-b970-596373fd0745\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-cfm6h"
Sep 17 17:57:57.214020 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.214001    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/184b0189-e1d0-4bfd-b970-596373fd0745-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-cfm6h\" (UID: \"184b0189-e1d0-4bfd-b970-596373fd0745\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-cfm6h"
Sep 17 17:57:57.227857 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.227833    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zxdd\" (UniqueName: \"kubernetes.io/projected/0cd09d92-625e-4b02-898e-0ab0fb807c58-kube-api-access-5zxdd\") pod \"cluster-samples-operator-685d7d68b5-wt46p\" (UID: \"0cd09d92-625e-4b02-898e-0ab0fb807c58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wt46p"
Sep 17 17:57:57.227985 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.227963    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6j58\" (UniqueName: \"kubernetes.io/projected/36a45fc1-bf32-4f4b-b741-75ac26197973-kube-api-access-f6j58\") pod \"volume-data-source-validator-5c4db8bdf9-rnprf\" (UID: \"36a45fc1-bf32-4f4b-b741-75ac26197973\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-rnprf"
Sep 17 17:57:57.228948 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.228929    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsm48\" (UniqueName: \"kubernetes.io/projected/184b0189-e1d0-4bfd-b970-596373fd0745-kube-api-access-hsm48\") pod \"kube-storage-version-migrator-operator-55889f7b47-cfm6h\" (UID: \"184b0189-e1d0-4bfd-b970-596373fd0745\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-cfm6h"
Sep 17 17:57:57.266166 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.266125    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-ztc5j"
Sep 17 17:57:57.313472 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.313441    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d396cd04-2841-4a84-8fd3-235da3979f1c-ca-trust-extracted\") pod \"image-registry-6787978d5c-jv2ww\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") " pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:57:57.313565 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.313489    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d396cd04-2841-4a84-8fd3-235da3979f1c-image-registry-private-configuration\") pod \"image-registry-6787978d5c-jv2ww\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") " pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:57:57.313565 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.313516    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-registry-tls\") pod \"image-registry-6787978d5c-jv2ww\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") " pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:57:57.313565 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.313544    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d396cd04-2841-4a84-8fd3-235da3979f1c-trusted-ca\") pod \"image-registry-6787978d5c-jv2ww\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") " pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:57:57.313701 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.313597    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7mcf\" (UniqueName: \"kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-kube-api-access-r7mcf\") pod \"image-registry-6787978d5c-jv2ww\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") " pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:57:57.313701 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:57.313668    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:57:57.313701 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:57.313690    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6787978d5c-jv2ww: secret "image-registry-tls" not found
Sep 17 17:57:57.313834 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:57.313759    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-registry-tls podName:d396cd04-2841-4a84-8fd3-235da3979f1c nodeName:}" failed. No retries permitted until 2026-09-17 17:57:57.813739406 +0000 UTC m=+59.914203219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-registry-tls") pod "image-registry-6787978d5c-jv2ww" (UID: "d396cd04-2841-4a84-8fd3-235da3979f1c") : secret "image-registry-tls" not found
Sep 17 17:57:57.313834 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.313814    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d396cd04-2841-4a84-8fd3-235da3979f1c-installation-pull-secrets\") pod \"image-registry-6787978d5c-jv2ww\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") " pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:57:57.313936 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.313925    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d396cd04-2841-4a84-8fd3-235da3979f1c-registry-certificates\") pod \"image-registry-6787978d5c-jv2ww\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") " pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:57:57.313983 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.313819    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d396cd04-2841-4a84-8fd3-235da3979f1c-ca-trust-extracted\") pod \"image-registry-6787978d5c-jv2ww\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") " pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:57:57.314033 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.313954    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-bound-sa-token\") pod \"image-registry-6787978d5c-jv2ww\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") " pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:57:57.314548 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.314525    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d396cd04-2841-4a84-8fd3-235da3979f1c-trusted-ca\") pod \"image-registry-6787978d5c-jv2ww\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") " pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:57:57.314681 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.314659    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d396cd04-2841-4a84-8fd3-235da3979f1c-registry-certificates\") pod \"image-registry-6787978d5c-jv2ww\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") " pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:57:57.316009 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.315988    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d396cd04-2841-4a84-8fd3-235da3979f1c-image-registry-private-configuration\") pod \"image-registry-6787978d5c-jv2ww\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") " pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:57:57.316092 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.316025    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d396cd04-2841-4a84-8fd3-235da3979f1c-installation-pull-secrets\") pod \"image-registry-6787978d5c-jv2ww\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") " pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:57:57.325379 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.325358    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7mcf\" (UniqueName: \"kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-kube-api-access-r7mcf\") pod \"image-registry-6787978d5c-jv2ww\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") " pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:57:57.327229 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.327207    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-bound-sa-token\") pod \"image-registry-6787978d5c-jv2ww\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") " pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:57:57.356709 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.356674    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-rnprf"
Sep 17 17:57:57.381197 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.381175    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-cfm6h"
Sep 17 17:57:57.560449 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.560412    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-ztc5j"]
Sep 17 17:57:57.563441 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:57:57.563416    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1080ecc5_cca9_47db_b949_05bdd3777351.slice/crio-05a2ee7835efe53fb7fd8a1ee88392ce760fd6c1cbc7a1a2ffab5897bba6f5a7 WatchSource:0}: Error finding container 05a2ee7835efe53fb7fd8a1ee88392ce760fd6c1cbc7a1a2ffab5897bba6f5a7: Status 404 returned error can't find the container with id 05a2ee7835efe53fb7fd8a1ee88392ce760fd6c1cbc7a1a2ffab5897bba6f5a7
Sep 17 17:57:57.657684 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.657660    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-rnprf"]
Sep 17 17:57:57.661496 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:57:57.661473    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36a45fc1_bf32_4f4b_b741_75ac26197973.slice/crio-7bfaf4fcb56fb6ce9392422010574a21b6b1619984e8c042346060eec2452e9e WatchSource:0}: Error finding container 7bfaf4fcb56fb6ce9392422010574a21b6b1619984e8c042346060eec2452e9e: Status 404 returned error can't find the container with id 7bfaf4fcb56fb6ce9392422010574a21b6b1619984e8c042346060eec2452e9e
Sep 17 17:57:57.691121 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.691098    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-cfm6h"]
Sep 17 17:57:57.694568 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:57:57.694548    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184b0189_e1d0_4bfd_b970_596373fd0745.slice/crio-d8010f4d0d62a96651ec65538d1ba8be13be7517e3302ee5522820420d9066ba WatchSource:0}: Error finding container d8010f4d0d62a96651ec65538d1ba8be13be7517e3302ee5522820420d9066ba: Status 404 returned error can't find the container with id d8010f4d0d62a96651ec65538d1ba8be13be7517e3302ee5522820420d9066ba
Sep 17 17:57:57.718105 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.718053    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0cd09d92-625e-4b02-898e-0ab0fb807c58-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-wt46p\" (UID: \"0cd09d92-625e-4b02-898e-0ab0fb807c58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wt46p"
Sep 17 17:57:57.718224 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:57.718208    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:57:57.718266 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:57.718260    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cd09d92-625e-4b02-898e-0ab0fb807c58-samples-operator-tls podName:0cd09d92-625e-4b02-898e-0ab0fb807c58 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:58.718247357 +0000 UTC m=+60.818711165 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0cd09d92-625e-4b02-898e-0ab0fb807c58-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-wt46p" (UID: "0cd09d92-625e-4b02-898e-0ab0fb807c58") : secret "samples-operator-tls" not found
Sep 17 17:57:57.819179 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:57.819136    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-registry-tls\") pod \"image-registry-6787978d5c-jv2ww\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") " pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:57:57.819274 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:57.819261    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:57:57.819309 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:57.819278    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6787978d5c-jv2ww: secret "image-registry-tls" not found
Sep 17 17:57:57.819420 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:57.819408    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-registry-tls podName:d396cd04-2841-4a84-8fd3-235da3979f1c nodeName:}" failed. No retries permitted until 2026-09-17 17:57:58.819389569 +0000 UTC m=+60.919853369 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-registry-tls") pod "image-registry-6787978d5c-jv2ww" (UID: "d396cd04-2841-4a84-8fd3-235da3979f1c") : secret "image-registry-tls" not found
Sep 17 17:57:58.284466 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:58.284425    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-ztc5j" event={"ID":"1080ecc5-cca9-47db-b949-05bdd3777351","Type":"ContainerStarted","Data":"05a2ee7835efe53fb7fd8a1ee88392ce760fd6c1cbc7a1a2ffab5897bba6f5a7"}
Sep 17 17:57:58.286221 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:58.286169    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-cfm6h" event={"ID":"184b0189-e1d0-4bfd-b970-596373fd0745","Type":"ContainerStarted","Data":"d8010f4d0d62a96651ec65538d1ba8be13be7517e3302ee5522820420d9066ba"}
Sep 17 17:57:58.287505 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:58.287478    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-rnprf" event={"ID":"36a45fc1-bf32-4f4b-b741-75ac26197973","Type":"ContainerStarted","Data":"7bfaf4fcb56fb6ce9392422010574a21b6b1619984e8c042346060eec2452e9e"}
Sep 17 17:57:58.713318 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:58.712212    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-j2gss"]
Sep 17 17:57:58.714570 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:58.714548    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-j2gss"
Sep 17 17:57:58.718649 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:58.718627    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-xhjdb\""
Sep 17 17:57:58.727555 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:58.727531    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s2g8\" (UniqueName: \"kubernetes.io/projected/1b091399-61b4-4444-a8c9-9d2849a7a2a9-kube-api-access-6s2g8\") pod \"network-check-source-7cbc45fddc-j2gss\" (UID: \"1b091399-61b4-4444-a8c9-9d2849a7a2a9\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-j2gss"
Sep 17 17:57:58.727672 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:58.727639    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0cd09d92-625e-4b02-898e-0ab0fb807c58-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-wt46p\" (UID: \"0cd09d92-625e-4b02-898e-0ab0fb807c58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wt46p"
Sep 17 17:57:58.727796 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:58.727782    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:57:58.727877 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:58.727840    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cd09d92-625e-4b02-898e-0ab0fb807c58-samples-operator-tls podName:0cd09d92-625e-4b02-898e-0ab0fb807c58 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:00.727823004 +0000 UTC m=+62.828286809 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0cd09d92-625e-4b02-898e-0ab0fb807c58-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-wt46p" (UID: "0cd09d92-625e-4b02-898e-0ab0fb807c58") : secret "samples-operator-tls" not found
Sep 17 17:57:58.731843 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:58.731818    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-j2gss"]
Sep 17 17:57:58.828605 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:58.828578    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s2g8\" (UniqueName: \"kubernetes.io/projected/1b091399-61b4-4444-a8c9-9d2849a7a2a9-kube-api-access-6s2g8\") pod \"network-check-source-7cbc45fddc-j2gss\" (UID: \"1b091399-61b4-4444-a8c9-9d2849a7a2a9\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-j2gss"
Sep 17 17:57:58.828708 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:58.828634    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-registry-tls\") pod \"image-registry-6787978d5c-jv2ww\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") " pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:57:58.828775 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:58.828759    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:57:58.828775 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:58.828769    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6787978d5c-jv2ww: secret "image-registry-tls" not found
Sep 17 17:57:58.828878 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:57:58.828817    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-registry-tls podName:d396cd04-2841-4a84-8fd3-235da3979f1c nodeName:}" failed. No retries permitted until 2026-09-17 17:58:00.828802288 +0000 UTC m=+62.929266087 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-registry-tls") pod "image-registry-6787978d5c-jv2ww" (UID: "d396cd04-2841-4a84-8fd3-235da3979f1c") : secret "image-registry-tls" not found
Sep 17 17:57:58.839072 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:58.839051    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s2g8\" (UniqueName: \"kubernetes.io/projected/1b091399-61b4-4444-a8c9-9d2849a7a2a9-kube-api-access-6s2g8\") pod \"network-check-source-7cbc45fddc-j2gss\" (UID: \"1b091399-61b4-4444-a8c9-9d2849a7a2a9\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-j2gss"
Sep 17 17:57:59.026663 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:59.026588    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-j2gss"
Sep 17 17:57:59.212472 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:59.212442    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qm8hx"
Sep 17 17:57:59.292576 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:59.292490    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-rnprf" event={"ID":"36a45fc1-bf32-4f4b-b741-75ac26197973","Type":"ContainerStarted","Data":"08800dccebaa57cab1865f192a10c9db1f169aca104fbb9295da5f277ce3cc19"}
Sep 17 17:57:59.311071 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:57:59.311014    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-rnprf" podStartSLOduration=1.174193203 podStartE2EDuration="2.310998509s" totalImagesPullingTime="1.136805306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:59.309550369 +0000 UTC m=+61.410014187" watchObservedRunningTime="2026-09-17 17:57:59.310998509 +0000 UTC m=+61.411462327"
Sep 17 17:58:00.235426 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:00.235342    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-j2gss"]
Sep 17 17:58:00.241671 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:58:00.241641    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b091399_61b4_4444_a8c9_9d2849a7a2a9.slice/crio-7b75a340221c7c7f0a47ac0f85e3be0517663e2ae71be135aed3dfb0c3e6d460 WatchSource:0}: Error finding container 7b75a340221c7c7f0a47ac0f85e3be0517663e2ae71be135aed3dfb0c3e6d460: Status 404 returned error can't find the container with id 7b75a340221c7c7f0a47ac0f85e3be0517663e2ae71be135aed3dfb0c3e6d460
Sep 17 17:58:00.296944 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:00.296910    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-j2gss" event={"ID":"1b091399-61b4-4444-a8c9-9d2849a7a2a9","Type":"ContainerStarted","Data":"7b75a340221c7c7f0a47ac0f85e3be0517663e2ae71be135aed3dfb0c3e6d460"}
Sep 17 17:58:00.298541 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:00.298519    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-ztc5j" event={"ID":"1080ecc5-cca9-47db-b949-05bdd3777351","Type":"ContainerStarted","Data":"0538f9bba6a5434158bd3c1f66b5ddf7507b9e4bfcf30414a27eaea89f21eea6"}
Sep 17 17:58:00.300063 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:00.300043    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-cfm6h" event={"ID":"184b0189-e1d0-4bfd-b970-596373fd0745","Type":"ContainerStarted","Data":"160752559f74b97eb0cb4dae6fc24dd395d56a89602288245ce49d905b5c9564"}
Sep 17 17:58:00.325041 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:00.325004    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-ztc5j" podStartSLOduration=1.933064554 podStartE2EDuration="4.324991176s" totalImagesPullingTime="2.391926622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:00.324388544 +0000 UTC m=+62.424852364" watchObservedRunningTime="2026-09-17 17:58:00.324991176 +0000 UTC m=+62.425454986"
Sep 17 17:58:00.347890 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:00.347843    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-cfm6h" podStartSLOduration=1.083023067 podStartE2EDuration="3.347826417s" totalImagesPullingTime="2.26480335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:00.346716801 +0000 UTC m=+62.447180621" watchObservedRunningTime="2026-09-17 17:58:00.347826417 +0000 UTC m=+62.448290237"
Sep 17 17:58:00.743489 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:00.743452    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0cd09d92-625e-4b02-898e-0ab0fb807c58-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-wt46p\" (UID: \"0cd09d92-625e-4b02-898e-0ab0fb807c58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wt46p"
Sep 17 17:58:00.743733 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:00.743693    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:58:00.743806 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:00.743785    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cd09d92-625e-4b02-898e-0ab0fb807c58-samples-operator-tls podName:0cd09d92-625e-4b02-898e-0ab0fb807c58 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:04.743765548 +0000 UTC m=+66.844229351 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0cd09d92-625e-4b02-898e-0ab0fb807c58-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-wt46p" (UID: "0cd09d92-625e-4b02-898e-0ab0fb807c58") : secret "samples-operator-tls" not found
Sep 17 17:58:00.844670 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:00.844635    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-registry-tls\") pod \"image-registry-6787978d5c-jv2ww\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") " pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:58:00.844882 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:00.844862    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:58:00.844981 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:00.844886    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6787978d5c-jv2ww: secret "image-registry-tls" not found
Sep 17 17:58:00.844981 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:00.844945    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-registry-tls podName:d396cd04-2841-4a84-8fd3-235da3979f1c nodeName:}" failed. No retries permitted until 2026-09-17 17:58:04.84492736 +0000 UTC m=+66.945391180 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-registry-tls") pod "image-registry-6787978d5c-jv2ww" (UID: "d396cd04-2841-4a84-8fd3-235da3979f1c") : secret "image-registry-tls" not found
Sep 17 17:58:01.838813 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:01.838775    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zdthk"]
Sep 17 17:58:01.841051 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:01.841024    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zdthk"
Sep 17 17:58:01.844516 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:01.844466    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 17:58:01.844679 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:01.844647    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2k5lz\""
Sep 17 17:58:01.845433 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:01.845413    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 17:58:01.857400 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:01.857380    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zdthk"]
Sep 17 17:58:01.954766 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:01.954731    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6efbbf7f-2529-4207-9138-8ff135899614-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdthk\" (UID: \"6efbbf7f-2529-4207-9138-8ff135899614\") " pod="openshift-insights/insights-runtime-extractor-zdthk"
Sep 17 17:58:01.954944 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:01.954782    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6efbbf7f-2529-4207-9138-8ff135899614-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdthk\" (UID: \"6efbbf7f-2529-4207-9138-8ff135899614\") " pod="openshift-insights/insights-runtime-extractor-zdthk"
Sep 17 17:58:01.954944 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:01.954874    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6efbbf7f-2529-4207-9138-8ff135899614-crio-socket\") pod \"insights-runtime-extractor-zdthk\" (UID: \"6efbbf7f-2529-4207-9138-8ff135899614\") " pod="openshift-insights/insights-runtime-extractor-zdthk"
Sep 17 17:58:01.954944 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:01.954919    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jqsk\" (UniqueName: \"kubernetes.io/projected/6efbbf7f-2529-4207-9138-8ff135899614-kube-api-access-6jqsk\") pod \"insights-runtime-extractor-zdthk\" (UID: \"6efbbf7f-2529-4207-9138-8ff135899614\") " pod="openshift-insights/insights-runtime-extractor-zdthk"
Sep 17 17:58:01.955098 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:01.955014    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6efbbf7f-2529-4207-9138-8ff135899614-data-volume\") pod \"insights-runtime-extractor-zdthk\" (UID: \"6efbbf7f-2529-4207-9138-8ff135899614\") " pod="openshift-insights/insights-runtime-extractor-zdthk"
Sep 17 17:58:02.057835 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:02.057806    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jqsk\" (UniqueName: \"kubernetes.io/projected/6efbbf7f-2529-4207-9138-8ff135899614-kube-api-access-6jqsk\") pod \"insights-runtime-extractor-zdthk\" (UID: \"6efbbf7f-2529-4207-9138-8ff135899614\") " pod="openshift-insights/insights-runtime-extractor-zdthk"
Sep 17 17:58:02.057993 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:02.057878    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6efbbf7f-2529-4207-9138-8ff135899614-data-volume\") pod \"insights-runtime-extractor-zdthk\" (UID: \"6efbbf7f-2529-4207-9138-8ff135899614\") " pod="openshift-insights/insights-runtime-extractor-zdthk"
Sep 17 17:58:02.057993 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:02.057916    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6efbbf7f-2529-4207-9138-8ff135899614-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdthk\" (UID: \"6efbbf7f-2529-4207-9138-8ff135899614\") " pod="openshift-insights/insights-runtime-extractor-zdthk"
Sep 17 17:58:02.057993 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:02.057943    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6efbbf7f-2529-4207-9138-8ff135899614-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdthk\" (UID: \"6efbbf7f-2529-4207-9138-8ff135899614\") " pod="openshift-insights/insights-runtime-extractor-zdthk"
Sep 17 17:58:02.058135 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:02.057996    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6efbbf7f-2529-4207-9138-8ff135899614-crio-socket\") pod \"insights-runtime-extractor-zdthk\" (UID: \"6efbbf7f-2529-4207-9138-8ff135899614\") " pod="openshift-insights/insights-runtime-extractor-zdthk"
Sep 17 17:58:02.058135 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:02.058077    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:58:02.058251 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:02.058143    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:02.058251 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:02.058214    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6efbbf7f-2529-4207-9138-8ff135899614-kube-rbac-proxy-cm podName:6efbbf7f-2529-4207-9138-8ff135899614 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:02.558190845 +0000 UTC m=+64.658654665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6efbbf7f-2529-4207-9138-8ff135899614-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zdthk" (UID: "6efbbf7f-2529-4207-9138-8ff135899614") : configmap "kube-rbac-proxy" not found
Sep 17 17:58:02.058251 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:02.058234    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6efbbf7f-2529-4207-9138-8ff135899614-insights-runtime-extractor-tls podName:6efbbf7f-2529-4207-9138-8ff135899614 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:02.558225775 +0000 UTC m=+64.658689575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6efbbf7f-2529-4207-9138-8ff135899614-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zdthk" (UID: "6efbbf7f-2529-4207-9138-8ff135899614") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:02.058251 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:02.058081    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6efbbf7f-2529-4207-9138-8ff135899614-crio-socket\") pod \"insights-runtime-extractor-zdthk\" (UID: \"6efbbf7f-2529-4207-9138-8ff135899614\") " pod="openshift-insights/insights-runtime-extractor-zdthk"
Sep 17 17:58:02.058409 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:02.058294    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6efbbf7f-2529-4207-9138-8ff135899614-data-volume\") pod \"insights-runtime-extractor-zdthk\" (UID: \"6efbbf7f-2529-4207-9138-8ff135899614\") " pod="openshift-insights/insights-runtime-extractor-zdthk"
Sep 17 17:58:02.068315 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:02.068288    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jqsk\" (UniqueName: \"kubernetes.io/projected/6efbbf7f-2529-4207-9138-8ff135899614-kube-api-access-6jqsk\") pod \"insights-runtime-extractor-zdthk\" (UID: \"6efbbf7f-2529-4207-9138-8ff135899614\") " pod="openshift-insights/insights-runtime-extractor-zdthk"
Sep 17 17:58:02.562527 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:02.562484    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6efbbf7f-2529-4207-9138-8ff135899614-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdthk\" (UID: \"6efbbf7f-2529-4207-9138-8ff135899614\") " pod="openshift-insights/insights-runtime-extractor-zdthk"
Sep 17 17:58:02.562527 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:02.562527    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6efbbf7f-2529-4207-9138-8ff135899614-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdthk\" (UID: \"6efbbf7f-2529-4207-9138-8ff135899614\") " pod="openshift-insights/insights-runtime-extractor-zdthk"
Sep 17 17:58:02.562756 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:02.562618    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:58:02.562756 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:02.562637    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:02.562756 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:02.562702    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6efbbf7f-2529-4207-9138-8ff135899614-insights-runtime-extractor-tls podName:6efbbf7f-2529-4207-9138-8ff135899614 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:03.562675845 +0000 UTC m=+65.663139650 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6efbbf7f-2529-4207-9138-8ff135899614-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zdthk" (UID: "6efbbf7f-2529-4207-9138-8ff135899614") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:02.562756 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:02.562730    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6efbbf7f-2529-4207-9138-8ff135899614-kube-rbac-proxy-cm podName:6efbbf7f-2529-4207-9138-8ff135899614 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:03.562719081 +0000 UTC m=+65.663182882 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6efbbf7f-2529-4207-9138-8ff135899614-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zdthk" (UID: "6efbbf7f-2529-4207-9138-8ff135899614") : configmap "kube-rbac-proxy" not found
Sep 17 17:58:03.311193 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:03.311093    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-j2gss" event={"ID":"1b091399-61b4-4444-a8c9-9d2849a7a2a9","Type":"ContainerStarted","Data":"e440e50bccc60127902a9be7178b65ae88740e2e77b50251791fe250befd2774"}
Sep 17 17:58:03.571315 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:03.571247    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6efbbf7f-2529-4207-9138-8ff135899614-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdthk\" (UID: \"6efbbf7f-2529-4207-9138-8ff135899614\") " pod="openshift-insights/insights-runtime-extractor-zdthk"
Sep 17 17:58:03.571315 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:03.571279    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6efbbf7f-2529-4207-9138-8ff135899614-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdthk\" (UID: \"6efbbf7f-2529-4207-9138-8ff135899614\") " pod="openshift-insights/insights-runtime-extractor-zdthk"
Sep 17 17:58:03.571474 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:03.571359    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:03.571474 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:03.571377    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:58:03.571474 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:03.571409    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6efbbf7f-2529-4207-9138-8ff135899614-insights-runtime-extractor-tls podName:6efbbf7f-2529-4207-9138-8ff135899614 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:05.57139772 +0000 UTC m=+67.671861518 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6efbbf7f-2529-4207-9138-8ff135899614-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zdthk" (UID: "6efbbf7f-2529-4207-9138-8ff135899614") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:03.571474 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:03.571436    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6efbbf7f-2529-4207-9138-8ff135899614-kube-rbac-proxy-cm podName:6efbbf7f-2529-4207-9138-8ff135899614 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:05.571419939 +0000 UTC m=+67.671883751 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6efbbf7f-2529-4207-9138-8ff135899614-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zdthk" (UID: "6efbbf7f-2529-4207-9138-8ff135899614") : configmap "kube-rbac-proxy" not found
Sep 17 17:58:03.690258 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:03.690210    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-j2gss" podStartSLOduration=2.932531292 podStartE2EDuration="5.690194669s" totalImagesPullingTime="2.757663377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:03.340859828 +0000 UTC m=+65.441323648" watchObservedRunningTime="2026-09-17 17:58:03.690194669 +0000 UTC m=+65.790658515"
Sep 17 17:58:03.690730 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:03.690716    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-2pp8d"]
Sep 17 17:58:03.692540 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:03.692526    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-2pp8d"
Sep 17 17:58:03.696093 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:03.696070    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 17 17:58:03.696218 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:03.696112    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-hbwr6\""
Sep 17 17:58:03.696502 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:03.696485    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 17 17:58:03.714065 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:03.714040    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-2pp8d"]
Sep 17 17:58:03.773007 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:03.772977    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/edd2cec8-35e2-4f8d-97db-99de69508531-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-2pp8d\" (UID: \"edd2cec8-35e2-4f8d-97db-99de69508531\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-2pp8d"
Sep 17 17:58:03.773120 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:03.773048    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/edd2cec8-35e2-4f8d-97db-99de69508531-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-2pp8d\" (UID: \"edd2cec8-35e2-4f8d-97db-99de69508531\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-2pp8d"
Sep 17 17:58:03.873643 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:03.873620    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/edd2cec8-35e2-4f8d-97db-99de69508531-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-2pp8d\" (UID: \"edd2cec8-35e2-4f8d-97db-99de69508531\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-2pp8d"
Sep 17 17:58:03.873775 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:03.873668    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/edd2cec8-35e2-4f8d-97db-99de69508531-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-2pp8d\" (UID: \"edd2cec8-35e2-4f8d-97db-99de69508531\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-2pp8d"
Sep 17 17:58:03.873775 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:03.873759    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:58:03.873874 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:03.873818    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edd2cec8-35e2-4f8d-97db-99de69508531-networking-console-plugin-cert podName:edd2cec8-35e2-4f8d-97db-99de69508531 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:04.373802854 +0000 UTC m=+66.474266651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/edd2cec8-35e2-4f8d-97db-99de69508531-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-2pp8d" (UID: "edd2cec8-35e2-4f8d-97db-99de69508531") : secret "networking-console-plugin-cert" not found
Sep 17 17:58:03.874313 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:03.874297    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/edd2cec8-35e2-4f8d-97db-99de69508531-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-2pp8d\" (UID: \"edd2cec8-35e2-4f8d-97db-99de69508531\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-2pp8d"
Sep 17 17:58:04.094210 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:04.094181    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-497mp"]
Sep 17 17:58:04.095906 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:04.095890    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-497mp"
Sep 17 17:58:04.098746 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:04.098724    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 17:58:04.098840 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:04.098795    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 17:58:04.099706 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:04.099472    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 17:58:04.099706 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:04.099543    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fvxxm\""
Sep 17 17:58:04.099706 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:04.099553    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 17:58:04.113808 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:04.113788    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-497mp"]
Sep 17 17:58:04.176038 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:04.175965    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b140d6b0-7532-452c-aa29-03ede624e935-signing-key\") pod \"service-ca-55f6fc7b86-497mp\" (UID: \"b140d6b0-7532-452c-aa29-03ede624e935\") " pod="openshift-service-ca/service-ca-55f6fc7b86-497mp"
Sep 17 17:58:04.176038 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:04.176002    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzpxk\" (UniqueName: \"kubernetes.io/projected/b140d6b0-7532-452c-aa29-03ede624e935-kube-api-access-rzpxk\") pod \"service-ca-55f6fc7b86-497mp\" (UID: \"b140d6b0-7532-452c-aa29-03ede624e935\") " pod="openshift-service-ca/service-ca-55f6fc7b86-497mp"
Sep 17 17:58:04.176038 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:04.176033    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b140d6b0-7532-452c-aa29-03ede624e935-signing-cabundle\") pod \"service-ca-55f6fc7b86-497mp\" (UID: \"b140d6b0-7532-452c-aa29-03ede624e935\") " pod="openshift-service-ca/service-ca-55f6fc7b86-497mp"
Sep 17 17:58:04.277183 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:04.277141    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b140d6b0-7532-452c-aa29-03ede624e935-signing-key\") pod \"service-ca-55f6fc7b86-497mp\" (UID: \"b140d6b0-7532-452c-aa29-03ede624e935\") " pod="openshift-service-ca/service-ca-55f6fc7b86-497mp"
Sep 17 17:58:04.277335 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:04.277189    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzpxk\" (UniqueName: \"kubernetes.io/projected/b140d6b0-7532-452c-aa29-03ede624e935-kube-api-access-rzpxk\") pod \"service-ca-55f6fc7b86-497mp\" (UID: \"b140d6b0-7532-452c-aa29-03ede624e935\") " pod="openshift-service-ca/service-ca-55f6fc7b86-497mp"
Sep 17 17:58:04.277335 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:04.277224    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b140d6b0-7532-452c-aa29-03ede624e935-signing-cabundle\") pod \"service-ca-55f6fc7b86-497mp\" (UID: \"b140d6b0-7532-452c-aa29-03ede624e935\") " pod="openshift-service-ca/service-ca-55f6fc7b86-497mp"
Sep 17 17:58:04.277959 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:04.277937    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b140d6b0-7532-452c-aa29-03ede624e935-signing-cabundle\") pod \"service-ca-55f6fc7b86-497mp\" (UID: \"b140d6b0-7532-452c-aa29-03ede624e935\") " pod="openshift-service-ca/service-ca-55f6fc7b86-497mp"
Sep 17 17:58:04.279696 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:04.279668    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b140d6b0-7532-452c-aa29-03ede624e935-signing-key\") pod \"service-ca-55f6fc7b86-497mp\" (UID: \"b140d6b0-7532-452c-aa29-03ede624e935\") " pod="openshift-service-ca/service-ca-55f6fc7b86-497mp"
Sep 17 17:58:04.286439 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:04.286421    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzpxk\" (UniqueName: \"kubernetes.io/projected/b140d6b0-7532-452c-aa29-03ede624e935-kube-api-access-rzpxk\") pod \"service-ca-55f6fc7b86-497mp\" (UID: \"b140d6b0-7532-452c-aa29-03ede624e935\") " pod="openshift-service-ca/service-ca-55f6fc7b86-497mp"
Sep 17 17:58:04.346835 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:04.346810    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2wq8v_6345bbfb-99aa-42ba-b5c9-d7c46ef2392f/dns-node-resolver/0.log"
Sep 17 17:58:04.378560 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:04.378530    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/edd2cec8-35e2-4f8d-97db-99de69508531-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-2pp8d\" (UID: \"edd2cec8-35e2-4f8d-97db-99de69508531\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-2pp8d"
Sep 17 17:58:04.378753 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:04.378737    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:58:04.378825 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:04.378816    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edd2cec8-35e2-4f8d-97db-99de69508531-networking-console-plugin-cert podName:edd2cec8-35e2-4f8d-97db-99de69508531 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:05.378799549 +0000 UTC m=+67.479263349 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/edd2cec8-35e2-4f8d-97db-99de69508531-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-2pp8d" (UID: "edd2cec8-35e2-4f8d-97db-99de69508531") : secret "networking-console-plugin-cert" not found
Sep 17 17:58:04.408717 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:04.408699    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-497mp"
Sep 17 17:58:04.693888 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:04.693860    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-497mp"]
Sep 17 17:58:04.697194 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:58:04.697163    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb140d6b0_7532_452c_aa29_03ede624e935.slice/crio-3992e36206977191cea09c43b7a62ba2c4fa4c1c7995244ca3c7751af2309d26 WatchSource:0}: Error finding container 3992e36206977191cea09c43b7a62ba2c4fa4c1c7995244ca3c7751af2309d26: Status 404 returned error can't find the container with id 3992e36206977191cea09c43b7a62ba2c4fa4c1c7995244ca3c7751af2309d26
Sep 17 17:58:04.782481 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:04.782410    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0cd09d92-625e-4b02-898e-0ab0fb807c58-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-wt46p\" (UID: \"0cd09d92-625e-4b02-898e-0ab0fb807c58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wt46p"
Sep 17 17:58:04.782603 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:04.782585    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:58:04.782667 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:04.782657    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cd09d92-625e-4b02-898e-0ab0fb807c58-samples-operator-tls podName:0cd09d92-625e-4b02-898e-0ab0fb807c58 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:12.782638309 +0000 UTC m=+74.883102113 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0cd09d92-625e-4b02-898e-0ab0fb807c58-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-wt46p" (UID: "0cd09d92-625e-4b02-898e-0ab0fb807c58") : secret "samples-operator-tls" not found
Sep 17 17:58:04.883743 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:04.883719    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-registry-tls\") pod \"image-registry-6787978d5c-jv2ww\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") " pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:58:04.883876 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:04.883858    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:58:04.883923 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:04.883879    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6787978d5c-jv2ww: secret "image-registry-tls" not found
Sep 17 17:58:04.883956 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:04.883925    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-registry-tls podName:d396cd04-2841-4a84-8fd3-235da3979f1c nodeName:}" failed. No retries permitted until 2026-09-17 17:58:12.883911671 +0000 UTC m=+74.984375468 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-registry-tls") pod "image-registry-6787978d5c-jv2ww" (UID: "d396cd04-2841-4a84-8fd3-235da3979f1c") : secret "image-registry-tls" not found
Sep 17 17:58:05.318404 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:05.318362    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-497mp" event={"ID":"b140d6b0-7532-452c-aa29-03ede624e935","Type":"ContainerStarted","Data":"3992e36206977191cea09c43b7a62ba2c4fa4c1c7995244ca3c7751af2309d26"}
Sep 17 17:58:05.388111 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:05.388077    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/edd2cec8-35e2-4f8d-97db-99de69508531-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-2pp8d\" (UID: \"edd2cec8-35e2-4f8d-97db-99de69508531\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-2pp8d"
Sep 17 17:58:05.388559 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:05.388255    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:58:05.388559 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:05.388350    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edd2cec8-35e2-4f8d-97db-99de69508531-networking-console-plugin-cert podName:edd2cec8-35e2-4f8d-97db-99de69508531 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:07.388328024 +0000 UTC m=+69.488791835 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/edd2cec8-35e2-4f8d-97db-99de69508531-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-2pp8d" (UID: "edd2cec8-35e2-4f8d-97db-99de69508531") : secret "networking-console-plugin-cert" not found
Sep 17 17:58:05.590413 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:05.590337    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6efbbf7f-2529-4207-9138-8ff135899614-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdthk\" (UID: \"6efbbf7f-2529-4207-9138-8ff135899614\") " pod="openshift-insights/insights-runtime-extractor-zdthk"
Sep 17 17:58:05.590561 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:05.590508    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6efbbf7f-2529-4207-9138-8ff135899614-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdthk\" (UID: \"6efbbf7f-2529-4207-9138-8ff135899614\") " pod="openshift-insights/insights-runtime-extractor-zdthk"
Sep 17 17:58:05.590561 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:05.590516    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:05.590634 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:05.590583    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6efbbf7f-2529-4207-9138-8ff135899614-insights-runtime-extractor-tls podName:6efbbf7f-2529-4207-9138-8ff135899614 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:09.590568377 +0000 UTC m=+71.691032174 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6efbbf7f-2529-4207-9138-8ff135899614-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zdthk" (UID: "6efbbf7f-2529-4207-9138-8ff135899614") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:05.590634 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:05.590588    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:58:05.590733 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:05.590638    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6efbbf7f-2529-4207-9138-8ff135899614-kube-rbac-proxy-cm podName:6efbbf7f-2529-4207-9138-8ff135899614 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:09.590622928 +0000 UTC m=+71.691086731 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6efbbf7f-2529-4207-9138-8ff135899614-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zdthk" (UID: "6efbbf7f-2529-4207-9138-8ff135899614") : configmap "kube-rbac-proxy" not found
Sep 17 17:58:05.691557 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:05.691518    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11b6807f-cb26-4117-b06d-eb974e5640ae-metrics-certs\") pod \"network-metrics-daemon-dsz4t\" (UID: \"11b6807f-cb26-4117-b06d-eb974e5640ae\") " pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:58:05.693900 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:05.693883    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:58:05.702573 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:05.702557    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:58:05.702629 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:05.702606    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11b6807f-cb26-4117-b06d-eb974e5640ae-metrics-certs podName:11b6807f-cb26-4117-b06d-eb974e5640ae nodeName:}" failed. No retries permitted until 2026-09-17 17:59:09.702591461 +0000 UTC m=+131.803055258 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11b6807f-cb26-4117-b06d-eb974e5640ae-metrics-certs") pod "network-metrics-daemon-dsz4t" (UID: "11b6807f-cb26-4117-b06d-eb974e5640ae") : secret "metrics-daemon-secret" not found
Sep 17 17:58:05.747202 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:05.747170    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k7mgr_383b592e-5a46-4299-9522-ac835c0adc61/node-ca/0.log"
Sep 17 17:58:05.792999 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:05.792967    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41ab02fa-694e-4b41-8969-0e5181664b3a-original-pull-secret\") pod \"global-pull-secret-syncer-shvtg\" (UID: \"41ab02fa-694e-4b41-8969-0e5181664b3a\") " pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:58:05.793212 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:05.793124    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2hn2\" (UniqueName: \"kubernetes.io/projected/c53ddf27-1765-4785-b122-1645ccc8d094-kube-api-access-p2hn2\") pod \"network-check-target-98c4r\" (UID: \"c53ddf27-1765-4785-b122-1645ccc8d094\") " pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:58:05.795498 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:05.795480    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:58:05.796682 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:05.796659    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2hn2\" (UniqueName: \"kubernetes.io/projected/c53ddf27-1765-4785-b122-1645ccc8d094-kube-api-access-p2hn2\") pod \"network-check-target-98c4r\" (UID: \"c53ddf27-1765-4785-b122-1645ccc8d094\") " pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:58:05.806203 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:05.806176    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41ab02fa-694e-4b41-8969-0e5181664b3a-original-pull-secret\") pod \"global-pull-secret-syncer-shvtg\" (UID: \"41ab02fa-694e-4b41-8969-0e5181664b3a\") " pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:58:05.953458 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:05.953425    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shvtg"
Sep 17 17:58:06.056492 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:06.056457    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6psft\""
Sep 17 17:58:06.065229 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:06.065210    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:58:06.348270 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:06.348186    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-p7h8g_47da4adc-a0aa-4908-91b7-d3baae40714d/serve-healthcheck-canary/0.log"
Sep 17 17:58:06.399351 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:06.399327    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1d64f49-582d-4389-b190-b84fae699d42-metrics-tls\") pod \"dns-default-jhhsj\" (UID: \"a1d64f49-582d-4389-b190-b84fae699d42\") " pod="openshift-dns/dns-default-jhhsj"
Sep 17 17:58:06.399691 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:06.399462    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:58:06.399691 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:06.399512    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1d64f49-582d-4389-b190-b84fae699d42-metrics-tls podName:a1d64f49-582d-4389-b190-b84fae699d42 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:38.39949749 +0000 UTC m=+100.499961290 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a1d64f49-582d-4389-b190-b84fae699d42-metrics-tls") pod "dns-default-jhhsj" (UID: "a1d64f49-582d-4389-b190-b84fae699d42") : secret "dns-default-metrics-tls" not found
Sep 17 17:58:06.581765 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:06.581692    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-98c4r"]
Sep 17 17:58:06.583702 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:06.583681    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-shvtg"]
Sep 17 17:58:06.585040 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:58:06.585018    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53ddf27_1765_4785_b122_1645ccc8d094.slice/crio-153203a21e335469276e4290e2b2c68bd7cae9f135f8b35a2aefdec11396c221 WatchSource:0}: Error finding container 153203a21e335469276e4290e2b2c68bd7cae9f135f8b35a2aefdec11396c221: Status 404 returned error can't find the container with id 153203a21e335469276e4290e2b2c68bd7cae9f135f8b35a2aefdec11396c221
Sep 17 17:58:06.586901 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:58:06.586885    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ab02fa_694e_4b41_8969_0e5181664b3a.slice/crio-058a2469ffe9de7439aeeb13a478776d705d6ae903b0e50565eb787229fc81a3 WatchSource:0}: Error finding container 058a2469ffe9de7439aeeb13a478776d705d6ae903b0e50565eb787229fc81a3: Status 404 returned error can't find the container with id 058a2469ffe9de7439aeeb13a478776d705d6ae903b0e50565eb787229fc81a3
Sep 17 17:58:07.151116 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:07.151082    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-cfm6h_184b0189-e1d0-4bfd-b970-596373fd0745/kube-storage-version-migrator-operator/0.log"
Sep 17 17:58:07.329853 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:07.329767    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-shvtg" event={"ID":"41ab02fa-694e-4b41-8969-0e5181664b3a","Type":"ContainerStarted","Data":"058a2469ffe9de7439aeeb13a478776d705d6ae903b0e50565eb787229fc81a3"}
Sep 17 17:58:07.331554 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:07.331523    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-98c4r" event={"ID":"c53ddf27-1765-4785-b122-1645ccc8d094","Type":"ContainerStarted","Data":"aef40fcc0732261862379a6f0ed9a117b10907da149cc003398da1143288d204"}
Sep 17 17:58:07.331692 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:07.331571    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-98c4r" event={"ID":"c53ddf27-1765-4785-b122-1645ccc8d094","Type":"ContainerStarted","Data":"153203a21e335469276e4290e2b2c68bd7cae9f135f8b35a2aefdec11396c221"}
Sep 17 17:58:07.331692 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:07.331625    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:58:07.333309 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:07.333285    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-497mp" event={"ID":"b140d6b0-7532-452c-aa29-03ede624e935","Type":"ContainerStarted","Data":"7da32c70c83ac57d19a79cc12eb8a5de7bfbe69d9ef521a149cc29ebf196cbed"}
Sep 17 17:58:07.363455 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:07.363405    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-98c4r" podStartSLOduration=67.363386926 podStartE2EDuration="1m7.363386926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:07.362383396 +0000 UTC m=+69.462847216" watchObservedRunningTime="2026-09-17 17:58:07.363386926 +0000 UTC m=+69.463850747"
Sep 17 17:58:07.390986 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:07.390934    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-497mp" podStartSLOduration=1.756723564 podStartE2EDuration="3.390917745s" totalImagesPullingTime="1.634194181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:07.388839334 +0000 UTC m=+69.489303154" watchObservedRunningTime="2026-09-17 17:58:07.390917745 +0000 UTC m=+69.491381564"
Sep 17 17:58:07.409128 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:07.409060    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/edd2cec8-35e2-4f8d-97db-99de69508531-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-2pp8d\" (UID: \"edd2cec8-35e2-4f8d-97db-99de69508531\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-2pp8d"
Sep 17 17:58:07.409516 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:07.409255    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:58:07.409516 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:07.409318    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edd2cec8-35e2-4f8d-97db-99de69508531-networking-console-plugin-cert podName:edd2cec8-35e2-4f8d-97db-99de69508531 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:11.409300427 +0000 UTC m=+73.509764244 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/edd2cec8-35e2-4f8d-97db-99de69508531-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-2pp8d" (UID: "edd2cec8-35e2-4f8d-97db-99de69508531") : secret "networking-console-plugin-cert" not found
Sep 17 17:58:09.632781 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:09.632746    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6efbbf7f-2529-4207-9138-8ff135899614-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdthk\" (UID: \"6efbbf7f-2529-4207-9138-8ff135899614\") " pod="openshift-insights/insights-runtime-extractor-zdthk"
Sep 17 17:58:09.633243 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:09.632798    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6efbbf7f-2529-4207-9138-8ff135899614-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdthk\" (UID: \"6efbbf7f-2529-4207-9138-8ff135899614\") " pod="openshift-insights/insights-runtime-extractor-zdthk"
Sep 17 17:58:09.633243 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:09.632883    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:58:09.633243 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:09.632923    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:09.633243 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:09.632969    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6efbbf7f-2529-4207-9138-8ff135899614-kube-rbac-proxy-cm podName:6efbbf7f-2529-4207-9138-8ff135899614 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:17.63294669 +0000 UTC m=+79.733410509 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6efbbf7f-2529-4207-9138-8ff135899614-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zdthk" (UID: "6efbbf7f-2529-4207-9138-8ff135899614") : configmap "kube-rbac-proxy" not found
Sep 17 17:58:09.633243 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:09.632988    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6efbbf7f-2529-4207-9138-8ff135899614-insights-runtime-extractor-tls podName:6efbbf7f-2529-4207-9138-8ff135899614 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:17.632978762 +0000 UTC m=+79.733442567 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6efbbf7f-2529-4207-9138-8ff135899614-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zdthk" (UID: "6efbbf7f-2529-4207-9138-8ff135899614") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:11.348832 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:11.348793    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-shvtg" event={"ID":"41ab02fa-694e-4b41-8969-0e5181664b3a","Type":"ContainerStarted","Data":"3bd2533bbcc46f6a829a5ed6a0d0dd4c68b817432862fe49a2a448444d280d53"}
Sep 17 17:58:11.445597 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:11.445562    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/edd2cec8-35e2-4f8d-97db-99de69508531-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-2pp8d\" (UID: \"edd2cec8-35e2-4f8d-97db-99de69508531\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-2pp8d"
Sep 17 17:58:11.445775 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:11.445720    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:58:11.445841 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:11.445790    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edd2cec8-35e2-4f8d-97db-99de69508531-networking-console-plugin-cert podName:edd2cec8-35e2-4f8d-97db-99de69508531 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:19.445766843 +0000 UTC m=+81.546230641 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/edd2cec8-35e2-4f8d-97db-99de69508531-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-2pp8d" (UID: "edd2cec8-35e2-4f8d-97db-99de69508531") : secret "networking-console-plugin-cert" not found
Sep 17 17:58:12.857714 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:12.857677    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0cd09d92-625e-4b02-898e-0ab0fb807c58-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-wt46p\" (UID: \"0cd09d92-625e-4b02-898e-0ab0fb807c58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wt46p"
Sep 17 17:58:12.860216 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:12.860192    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0cd09d92-625e-4b02-898e-0ab0fb807c58-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-wt46p\" (UID: \"0cd09d92-625e-4b02-898e-0ab0fb807c58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wt46p"
Sep 17 17:58:12.958699 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:12.958674    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-registry-tls\") pod \"image-registry-6787978d5c-jv2ww\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") " pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:58:12.961112 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:12.961077    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-registry-tls\") pod \"image-registry-6787978d5c-jv2ww\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") " pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:58:13.006471 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:13.006450    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4g6xq\""
Sep 17 17:58:13.015228 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:13.015209    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wt46p"
Sep 17 17:58:13.062914 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:13.062891    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lg89h\""
Sep 17 17:58:13.071338 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:13.071317    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:58:13.294224 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:13.294178    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-shvtg" podStartSLOduration=69.336467727 podStartE2EDuration="1m13.294145494s" totalImagesPullingTime="3.957677767s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:11.372488026 +0000 UTC m=+73.472951841" watchObservedRunningTime="2026-09-17 17:58:13.294145494 +0000 UTC m=+75.394609313"
Sep 17 17:58:13.294656 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:13.294638    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wt46p"]
Sep 17 17:58:13.358029 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:13.357988    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wt46p" event={"ID":"0cd09d92-625e-4b02-898e-0ab0fb807c58","Type":"ContainerStarted","Data":"5217d4f59767187e25f771fd2a475e114738aca74d1bc02eee119af11422be8e"}
Sep 17 17:58:13.360849 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:13.360822    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6787978d5c-jv2ww"]
Sep 17 17:58:13.363315 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:58:13.363278    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd396cd04_2841_4a84_8fd3_235da3979f1c.slice/crio-98e0f0d736e30e6ab945ada946877c9ecd1f3b920af88cb49a939b3d765bde9b WatchSource:0}: Error finding container 98e0f0d736e30e6ab945ada946877c9ecd1f3b920af88cb49a939b3d765bde9b: Status 404 returned error can't find the container with id 98e0f0d736e30e6ab945ada946877c9ecd1f3b920af88cb49a939b3d765bde9b
Sep 17 17:58:14.363788 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:14.363752    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6787978d5c-jv2ww" event={"ID":"d396cd04-2841-4a84-8fd3-235da3979f1c","Type":"ContainerStarted","Data":"4fbfa7904387c10692acdafbffc0a7e13a139ba62ee8498d0bc91598ff7703d6"}
Sep 17 17:58:14.364277 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:14.363811    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6787978d5c-jv2ww" event={"ID":"d396cd04-2841-4a84-8fd3-235da3979f1c","Type":"ContainerStarted","Data":"98e0f0d736e30e6ab945ada946877c9ecd1f3b920af88cb49a939b3d765bde9b"}
Sep 17 17:58:14.364277 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:14.363881    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:58:14.388840 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:14.388793    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6787978d5c-jv2ww" podStartSLOduration=17.388772983 podStartE2EDuration="17.388772983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:14.38786156 +0000 UTC m=+76.488325380" watchObservedRunningTime="2026-09-17 17:58:14.388772983 +0000 UTC m=+76.489236796"
Sep 17 17:58:15.368698 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:15.368665    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wt46p" event={"ID":"0cd09d92-625e-4b02-898e-0ab0fb807c58","Type":"ContainerStarted","Data":"3c9683eeabfc4a740ef078f51fefc79366a6637ec75e578fec2c8a04e1b02b5a"}
Sep 17 17:58:15.369138 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:15.368720    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wt46p" event={"ID":"0cd09d92-625e-4b02-898e-0ab0fb807c58","Type":"ContainerStarted","Data":"13b45ebab5a2100efaaac3a76ba70c6811f63025c5b738947450409c54f26454"}
Sep 17 17:58:15.391165 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:15.391101    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wt46p" podStartSLOduration=16.777486774 podStartE2EDuration="18.391087393s" totalImagesPullingTime="1.613600619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:15.390183112 +0000 UTC m=+77.490646930" watchObservedRunningTime="2026-09-17 17:58:15.391087393 +0000 UTC m=+77.491551210"
Sep 17 17:58:17.698302 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:17.698266    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6efbbf7f-2529-4207-9138-8ff135899614-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdthk\" (UID: \"6efbbf7f-2529-4207-9138-8ff135899614\") " pod="openshift-insights/insights-runtime-extractor-zdthk"
Sep 17 17:58:17.698302 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:17.698304    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6efbbf7f-2529-4207-9138-8ff135899614-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdthk\" (UID: \"6efbbf7f-2529-4207-9138-8ff135899614\") " pod="openshift-insights/insights-runtime-extractor-zdthk"
Sep 17 17:58:17.698861 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:17.698841    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6efbbf7f-2529-4207-9138-8ff135899614-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdthk\" (UID: \"6efbbf7f-2529-4207-9138-8ff135899614\") " pod="openshift-insights/insights-runtime-extractor-zdthk"
Sep 17 17:58:17.700733 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:17.700714    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6efbbf7f-2529-4207-9138-8ff135899614-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdthk\" (UID: \"6efbbf7f-2529-4207-9138-8ff135899614\") " pod="openshift-insights/insights-runtime-extractor-zdthk"
Sep 17 17:58:17.751908 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:17.751886    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zdthk"
Sep 17 17:58:18.025855 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:18.025784    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zdthk"]
Sep 17 17:58:18.029001 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:58:18.028973    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6efbbf7f_2529_4207_9138_8ff135899614.slice/crio-314fb7b4a9593aeef68a44aead7005b215a7d84459530889a37134dc4ccc9221 WatchSource:0}: Error finding container 314fb7b4a9593aeef68a44aead7005b215a7d84459530889a37134dc4ccc9221: Status 404 returned error can't find the container with id 314fb7b4a9593aeef68a44aead7005b215a7d84459530889a37134dc4ccc9221
Sep 17 17:58:18.384585 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:18.384554    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zdthk" event={"ID":"6efbbf7f-2529-4207-9138-8ff135899614","Type":"ContainerStarted","Data":"bd85d0f17b7db3246990bfaa84eb8e591920572b58b1f28ff562d2c6457aab77"}
Sep 17 17:58:18.384739 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:18.384605    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zdthk" event={"ID":"6efbbf7f-2529-4207-9138-8ff135899614","Type":"ContainerStarted","Data":"314fb7b4a9593aeef68a44aead7005b215a7d84459530889a37134dc4ccc9221"}
Sep 17 17:58:19.389933 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:19.389847    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zdthk" event={"ID":"6efbbf7f-2529-4207-9138-8ff135899614","Type":"ContainerStarted","Data":"e8020ae0c472f48a88c95e6d2419077987e9f508a18c3436f75e8ad45dd75d2c"}
Sep 17 17:58:19.512863 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:19.512826    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/edd2cec8-35e2-4f8d-97db-99de69508531-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-2pp8d\" (UID: \"edd2cec8-35e2-4f8d-97db-99de69508531\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-2pp8d"
Sep 17 17:58:19.515553 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:19.515522    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/edd2cec8-35e2-4f8d-97db-99de69508531-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-2pp8d\" (UID: \"edd2cec8-35e2-4f8d-97db-99de69508531\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-2pp8d"
Sep 17 17:58:19.600464 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:19.600433    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-2pp8d"
Sep 17 17:58:20.191896 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:20.191761    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-2pp8d"]
Sep 17 17:58:20.195552 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:58:20.195529    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd2cec8_35e2_4f8d_97db_99de69508531.slice/crio-fddef263e5c762d518567a59aeba09ee6c64dbd7e143e38e9fed312e29aa638a WatchSource:0}: Error finding container fddef263e5c762d518567a59aeba09ee6c64dbd7e143e38e9fed312e29aa638a: Status 404 returned error can't find the container with id fddef263e5c762d518567a59aeba09ee6c64dbd7e143e38e9fed312e29aa638a
Sep 17 17:58:20.398222 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:20.398181    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zdthk" event={"ID":"6efbbf7f-2529-4207-9138-8ff135899614","Type":"ContainerStarted","Data":"0814b8c7276967f6e4cd7b041b9f7da409ade05e998facca407a1e3464b31f58"}
Sep 17 17:58:20.399421 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:20.399400    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-2pp8d" event={"ID":"edd2cec8-35e2-4f8d-97db-99de69508531","Type":"ContainerStarted","Data":"fddef263e5c762d518567a59aeba09ee6c64dbd7e143e38e9fed312e29aa638a"}
Sep 17 17:58:21.404425 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:21.404393    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-2pp8d" event={"ID":"edd2cec8-35e2-4f8d-97db-99de69508531","Type":"ContainerStarted","Data":"a15b86efd6bac129e4dd4a2be7299e8f3d127e2370d0f61a10eabb9a4c3e7dd8"}
Sep 17 17:58:21.424902 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:21.424821    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zdthk" podStartSLOduration=18.594486411 podStartE2EDuration="20.424804317s" totalImagesPullingTime="1.830317906s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:58:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:20.441373518 +0000 UTC m=+82.541837328" watchObservedRunningTime="2026-09-17 17:58:21.424804317 +0000 UTC m=+83.525268136"
Sep 17 17:58:21.425637 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:21.425610    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-2pp8d" podStartSLOduration=17.38161877 podStartE2EDuration="18.42559856s" totalImagesPullingTime="1.04397979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:21.423980459 +0000 UTC m=+83.524444277" watchObservedRunningTime="2026-09-17 17:58:21.42559856 +0000 UTC m=+83.526062379"
Sep 17 17:58:28.635787 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:28.635752    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6787978d5c-jv2ww"]
Sep 17 17:58:28.639921 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:28.639894    2581 patch_prober.go:28] interesting pod/image-registry-6787978d5c-jv2ww container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 17:58:28.640056 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:28.639946    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6787978d5c-jv2ww" podUID="d396cd04-2841-4a84-8fd3-235da3979f1c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:58:28.807758 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:28.807721    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56d8969c4d-qgjx4"]
Sep 17 17:58:28.812014 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:28.811993    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4"
Sep 17 17:58:28.850521 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:28.850495    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56d8969c4d-qgjx4"]
Sep 17 17:58:28.983320 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:28.983258    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ad48dd9-2592-40af-822e-8b71a26febfc-trusted-ca\") pod \"image-registry-56d8969c4d-qgjx4\" (UID: \"3ad48dd9-2592-40af-822e-8b71a26febfc\") " pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4"
Sep 17 17:58:28.983320 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:28.983294    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ad48dd9-2592-40af-822e-8b71a26febfc-bound-sa-token\") pod \"image-registry-56d8969c4d-qgjx4\" (UID: \"3ad48dd9-2592-40af-822e-8b71a26febfc\") " pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4"
Sep 17 17:58:28.983533 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:28.983320    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ad48dd9-2592-40af-822e-8b71a26febfc-image-registry-private-configuration\") pod \"image-registry-56d8969c4d-qgjx4\" (UID: \"3ad48dd9-2592-40af-822e-8b71a26febfc\") " pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4"
Sep 17 17:58:28.983533 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:28.983344    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ad48dd9-2592-40af-822e-8b71a26febfc-registry-tls\") pod \"image-registry-56d8969c4d-qgjx4\" (UID: \"3ad48dd9-2592-40af-822e-8b71a26febfc\") " pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4"
Sep 17 17:58:28.983533 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:28.983359    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqjb9\" (UniqueName: \"kubernetes.io/projected/3ad48dd9-2592-40af-822e-8b71a26febfc-kube-api-access-dqjb9\") pod \"image-registry-56d8969c4d-qgjx4\" (UID: \"3ad48dd9-2592-40af-822e-8b71a26febfc\") " pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4"
Sep 17 17:58:28.983533 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:28.983426    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ad48dd9-2592-40af-822e-8b71a26febfc-registry-certificates\") pod \"image-registry-56d8969c4d-qgjx4\" (UID: \"3ad48dd9-2592-40af-822e-8b71a26febfc\") " pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4"
Sep 17 17:58:28.983533 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:28.983492    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ad48dd9-2592-40af-822e-8b71a26febfc-ca-trust-extracted\") pod \"image-registry-56d8969c4d-qgjx4\" (UID: \"3ad48dd9-2592-40af-822e-8b71a26febfc\") " pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4"
Sep 17 17:58:28.983533 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:28.983523    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ad48dd9-2592-40af-822e-8b71a26febfc-installation-pull-secrets\") pod \"image-registry-56d8969c4d-qgjx4\" (UID: \"3ad48dd9-2592-40af-822e-8b71a26febfc\") " pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4"
Sep 17 17:58:29.084798 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:29.084768    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ad48dd9-2592-40af-822e-8b71a26febfc-bound-sa-token\") pod \"image-registry-56d8969c4d-qgjx4\" (UID: \"3ad48dd9-2592-40af-822e-8b71a26febfc\") " pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4"
Sep 17 17:58:29.084798 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:29.084798    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ad48dd9-2592-40af-822e-8b71a26febfc-image-registry-private-configuration\") pod \"image-registry-56d8969c4d-qgjx4\" (UID: \"3ad48dd9-2592-40af-822e-8b71a26febfc\") " pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4"
Sep 17 17:58:29.084964 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:29.084815    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ad48dd9-2592-40af-822e-8b71a26febfc-registry-tls\") pod \"image-registry-56d8969c4d-qgjx4\" (UID: \"3ad48dd9-2592-40af-822e-8b71a26febfc\") " pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4"
Sep 17 17:58:29.084964 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:29.084829    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqjb9\" (UniqueName: \"kubernetes.io/projected/3ad48dd9-2592-40af-822e-8b71a26febfc-kube-api-access-dqjb9\") pod \"image-registry-56d8969c4d-qgjx4\" (UID: \"3ad48dd9-2592-40af-822e-8b71a26febfc\") " pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4"
Sep 17 17:58:29.084964 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:29.084849    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ad48dd9-2592-40af-822e-8b71a26febfc-registry-certificates\") pod \"image-registry-56d8969c4d-qgjx4\" (UID: \"3ad48dd9-2592-40af-822e-8b71a26febfc\") " pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4"
Sep 17 17:58:29.084964 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:29.084887    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ad48dd9-2592-40af-822e-8b71a26febfc-ca-trust-extracted\") pod \"image-registry-56d8969c4d-qgjx4\" (UID: \"3ad48dd9-2592-40af-822e-8b71a26febfc\") " pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4"
Sep 17 17:58:29.085114 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:29.084990    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ad48dd9-2592-40af-822e-8b71a26febfc-installation-pull-secrets\") pod \"image-registry-56d8969c4d-qgjx4\" (UID: \"3ad48dd9-2592-40af-822e-8b71a26febfc\") " pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4"
Sep 17 17:58:29.085114 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:29.085069    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ad48dd9-2592-40af-822e-8b71a26febfc-trusted-ca\") pod \"image-registry-56d8969c4d-qgjx4\" (UID: \"3ad48dd9-2592-40af-822e-8b71a26febfc\") " pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4"
Sep 17 17:58:29.085614 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:29.085589    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ad48dd9-2592-40af-822e-8b71a26febfc-ca-trust-extracted\") pod \"image-registry-56d8969c4d-qgjx4\" (UID: \"3ad48dd9-2592-40af-822e-8b71a26febfc\") " pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4"
Sep 17 17:58:29.086004 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:29.085982    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ad48dd9-2592-40af-822e-8b71a26febfc-registry-certificates\") pod \"image-registry-56d8969c4d-qgjx4\" (UID: \"3ad48dd9-2592-40af-822e-8b71a26febfc\") " pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4"
Sep 17 17:58:29.086199 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:29.086179    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ad48dd9-2592-40af-822e-8b71a26febfc-trusted-ca\") pod \"image-registry-56d8969c4d-qgjx4\" (UID: \"3ad48dd9-2592-40af-822e-8b71a26febfc\") " pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4"
Sep 17 17:58:29.087581 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:29.087551    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ad48dd9-2592-40af-822e-8b71a26febfc-image-registry-private-configuration\") pod \"image-registry-56d8969c4d-qgjx4\" (UID: \"3ad48dd9-2592-40af-822e-8b71a26febfc\") " pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4"
Sep 17 17:58:29.087581 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:29.087570    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ad48dd9-2592-40af-822e-8b71a26febfc-registry-tls\") pod \"image-registry-56d8969c4d-qgjx4\" (UID: \"3ad48dd9-2592-40af-822e-8b71a26febfc\") " pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4"
Sep 17 17:58:29.087696 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:29.087605    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ad48dd9-2592-40af-822e-8b71a26febfc-installation-pull-secrets\") pod \"image-registry-56d8969c4d-qgjx4\" (UID: \"3ad48dd9-2592-40af-822e-8b71a26febfc\") " pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4"
Sep 17 17:58:29.114744 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:29.114718    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqjb9\" (UniqueName: \"kubernetes.io/projected/3ad48dd9-2592-40af-822e-8b71a26febfc-kube-api-access-dqjb9\") pod \"image-registry-56d8969c4d-qgjx4\" (UID: \"3ad48dd9-2592-40af-822e-8b71a26febfc\") " pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4"
Sep 17 17:58:29.115258 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:29.115238    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ad48dd9-2592-40af-822e-8b71a26febfc-bound-sa-token\") pod \"image-registry-56d8969c4d-qgjx4\" (UID: \"3ad48dd9-2592-40af-822e-8b71a26febfc\") " pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4"
Sep 17 17:58:29.120192 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:29.120175    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4"
Sep 17 17:58:29.410437 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:29.410357    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56d8969c4d-qgjx4"]
Sep 17 17:58:29.412350 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:58:29.412327    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ad48dd9_2592_40af_822e_8b71a26febfc.slice/crio-420f00c8b7eadd074b780d8f12531f796ace72955c79bbee3e02aecd1ab29457 WatchSource:0}: Error finding container 420f00c8b7eadd074b780d8f12531f796ace72955c79bbee3e02aecd1ab29457: Status 404 returned error can't find the container with id 420f00c8b7eadd074b780d8f12531f796ace72955c79bbee3e02aecd1ab29457
Sep 17 17:58:29.430254 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:29.430230    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4" event={"ID":"3ad48dd9-2592-40af-822e-8b71a26febfc","Type":"ContainerStarted","Data":"420f00c8b7eadd074b780d8f12531f796ace72955c79bbee3e02aecd1ab29457"}
Sep 17 17:58:30.435664 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:30.435630    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4" event={"ID":"3ad48dd9-2592-40af-822e-8b71a26febfc","Type":"ContainerStarted","Data":"d767727f0ae40badde66283531b0fbd84dbab77ce574f380be31ca16ee285392"}
Sep 17 17:58:30.436025 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:30.435752    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4"
Sep 17 17:58:36.250197 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.249914    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4" podStartSLOduration=8.249893679 podStartE2EDuration="8.249893679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:30.464568299 +0000 UTC m=+92.565032116" watchObservedRunningTime="2026-09-17 17:58:36.249893679 +0000 UTC m=+98.350357498"
Sep 17 17:58:36.251346 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.251319    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-595f6ff97f-8xv29"]
Sep 17 17:58:36.302611 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.302579    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-595f6ff97f-8xv29"]
Sep 17 17:58:36.302737 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.302683    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595f6ff97f-8xv29"
Sep 17 17:58:36.307449 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.307422    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 17:58:36.307555 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.307422    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 17:58:36.308009 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.307992    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 17:58:36.308720 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.308655    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 17:58:36.308720 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.308669    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 17:58:36.308920 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.308802    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 17:58:36.309405 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.309027    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-h8dtw\""
Sep 17 17:58:36.310091 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.310076    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 17:58:36.313911 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.313891    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 17:58:36.436547 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.436522    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-trusted-ca-bundle\") pod \"console-595f6ff97f-8xv29\" (UID: \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\") " pod="openshift-console/console-595f6ff97f-8xv29"
Sep 17 17:58:36.436669 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.436564    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-console-oauth-config\") pod \"console-595f6ff97f-8xv29\" (UID: \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\") " pod="openshift-console/console-595f6ff97f-8xv29"
Sep 17 17:58:36.436669 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.436580    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-service-ca\") pod \"console-595f6ff97f-8xv29\" (UID: \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\") " pod="openshift-console/console-595f6ff97f-8xv29"
Sep 17 17:58:36.436779 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.436671    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-console-serving-cert\") pod \"console-595f6ff97f-8xv29\" (UID: \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\") " pod="openshift-console/console-595f6ff97f-8xv29"
Sep 17 17:58:36.436779 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.436730    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-oauth-serving-cert\") pod \"console-595f6ff97f-8xv29\" (UID: \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\") " pod="openshift-console/console-595f6ff97f-8xv29"
Sep 17 17:58:36.436881 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.436781    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m2j9\" (UniqueName: \"kubernetes.io/projected/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-kube-api-access-7m2j9\") pod \"console-595f6ff97f-8xv29\" (UID: \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\") " pod="openshift-console/console-595f6ff97f-8xv29"
Sep 17 17:58:36.436881 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.436817    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-console-config\") pod \"console-595f6ff97f-8xv29\" (UID: \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\") " pod="openshift-console/console-595f6ff97f-8xv29"
Sep 17 17:58:36.537412 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.537357    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-console-serving-cert\") pod \"console-595f6ff97f-8xv29\" (UID: \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\") " pod="openshift-console/console-595f6ff97f-8xv29"
Sep 17 17:58:36.537412 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.537385    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-oauth-serving-cert\") pod \"console-595f6ff97f-8xv29\" (UID: \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\") " pod="openshift-console/console-595f6ff97f-8xv29"
Sep 17 17:58:36.537412 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.537407    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m2j9\" (UniqueName: \"kubernetes.io/projected/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-kube-api-access-7m2j9\") pod \"console-595f6ff97f-8xv29\" (UID: \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\") " pod="openshift-console/console-595f6ff97f-8xv29"
Sep 17 17:58:36.537609 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.537436    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-console-config\") pod \"console-595f6ff97f-8xv29\" (UID: \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\") " pod="openshift-console/console-595f6ff97f-8xv29"
Sep 17 17:58:36.537609 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.537479    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-trusted-ca-bundle\") pod \"console-595f6ff97f-8xv29\" (UID: \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\") " pod="openshift-console/console-595f6ff97f-8xv29"
Sep 17 17:58:36.537609 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.537525    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-console-oauth-config\") pod \"console-595f6ff97f-8xv29\" (UID: \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\") " pod="openshift-console/console-595f6ff97f-8xv29"
Sep 17 17:58:36.537609 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.537550    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-service-ca\") pod \"console-595f6ff97f-8xv29\" (UID: \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\") " pod="openshift-console/console-595f6ff97f-8xv29"
Sep 17 17:58:36.538328 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.538228    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-console-config\") pod \"console-595f6ff97f-8xv29\" (UID: \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\") " pod="openshift-console/console-595f6ff97f-8xv29"
Sep 17 17:58:36.538328 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.538230    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-service-ca\") pod \"console-595f6ff97f-8xv29\" (UID: \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\") " pod="openshift-console/console-595f6ff97f-8xv29"
Sep 17 17:58:36.538658 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.538634    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-trusted-ca-bundle\") pod \"console-595f6ff97f-8xv29\" (UID: \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\") " pod="openshift-console/console-595f6ff97f-8xv29"
Sep 17 17:58:36.538744 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.538670    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-oauth-serving-cert\") pod \"console-595f6ff97f-8xv29\" (UID: \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\") " pod="openshift-console/console-595f6ff97f-8xv29"
Sep 17 17:58:36.539938 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.539919    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-console-oauth-config\") pod \"console-595f6ff97f-8xv29\" (UID: \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\") " pod="openshift-console/console-595f6ff97f-8xv29"
Sep 17 17:58:36.540100 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.540084    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-console-serving-cert\") pod \"console-595f6ff97f-8xv29\" (UID: \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\") " pod="openshift-console/console-595f6ff97f-8xv29"
Sep 17 17:58:36.547282 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.547264    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m2j9\" (UniqueName: \"kubernetes.io/projected/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-kube-api-access-7m2j9\") pod \"console-595f6ff97f-8xv29\" (UID: \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\") " pod="openshift-console/console-595f6ff97f-8xv29"
Sep 17 17:58:36.611528 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.611503    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595f6ff97f-8xv29"
Sep 17 17:58:36.916788 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:36.916758    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-595f6ff97f-8xv29"]
Sep 17 17:58:36.919912 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:58:36.919891    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a3be75_f0e0_4691_8365_bf0b19b2aeb2.slice/crio-30858cabe95cea21928b78000d8982d27c05910514fe57e9836be8640be83e7f WatchSource:0}: Error finding container 30858cabe95cea21928b78000d8982d27c05910514fe57e9836be8640be83e7f: Status 404 returned error can't find the container with id 30858cabe95cea21928b78000d8982d27c05910514fe57e9836be8640be83e7f
Sep 17 17:58:37.460110 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:37.460079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595f6ff97f-8xv29" event={"ID":"01a3be75-f0e0-4691-8365-bf0b19b2aeb2","Type":"ContainerStarted","Data":"30858cabe95cea21928b78000d8982d27c05910514fe57e9836be8640be83e7f"}
Sep 17 17:58:38.341760 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:38.341228    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-98c4r"
Sep 17 17:58:38.452303 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:38.452264    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1d64f49-582d-4389-b190-b84fae699d42-metrics-tls\") pod \"dns-default-jhhsj\" (UID: \"a1d64f49-582d-4389-b190-b84fae699d42\") " pod="openshift-dns/dns-default-jhhsj"
Sep 17 17:58:38.455097 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:38.455069    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1d64f49-582d-4389-b190-b84fae699d42-metrics-tls\") pod \"dns-default-jhhsj\" (UID: \"a1d64f49-582d-4389-b190-b84fae699d42\") " pod="openshift-dns/dns-default-jhhsj"
Sep 17 17:58:38.641214 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:38.641190    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:58:38.709299 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:38.709272    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zr8ck\""
Sep 17 17:58:38.717410 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:38.717373    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jhhsj"
Sep 17 17:58:41.131308 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:41.131279    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jhhsj"]
Sep 17 17:58:41.134100 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:58:41.134076    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d64f49_582d_4389_b190_b84fae699d42.slice/crio-ca0558e6ef98faa2d94a10daef562bd84309c37f335dd34bef6eb1d4c933c9f1 WatchSource:0}: Error finding container ca0558e6ef98faa2d94a10daef562bd84309c37f335dd34bef6eb1d4c933c9f1: Status 404 returned error can't find the container with id ca0558e6ef98faa2d94a10daef562bd84309c37f335dd34bef6eb1d4c933c9f1
Sep 17 17:58:41.476331 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:41.476244    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595f6ff97f-8xv29" event={"ID":"01a3be75-f0e0-4691-8365-bf0b19b2aeb2","Type":"ContainerStarted","Data":"38ff75e4bfd711cca31205f2faa1b2c15dc3c21fd93eb56c23ed16eb3ddf65f2"}
Sep 17 17:58:41.477397 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:41.477371    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jhhsj" event={"ID":"a1d64f49-582d-4389-b190-b84fae699d42","Type":"ContainerStarted","Data":"ca0558e6ef98faa2d94a10daef562bd84309c37f335dd34bef6eb1d4c933c9f1"}
Sep 17 17:58:43.485866 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:43.485830    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jhhsj" event={"ID":"a1d64f49-582d-4389-b190-b84fae699d42","Type":"ContainerStarted","Data":"a62b175b32849f9a6560a4d2ff21585c9d3e9d952b4feebe8aadd36555a9f797"}
Sep 17 17:58:43.486264 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:43.485912    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jhhsj" event={"ID":"a1d64f49-582d-4389-b190-b84fae699d42","Type":"ContainerStarted","Data":"f2538eaaf2f4e0fa39243a21bf1f3b3dfb534e7a479b778b88bd7c91477567dc"}
Sep 17 17:58:43.486264 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:43.485960    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jhhsj"
Sep 17 17:58:43.512627 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:43.512579    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-595f6ff97f-8xv29" podStartSLOduration=3.5146978840000003 podStartE2EDuration="7.512560588s" totalImagesPullingTime="3.997862704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:41.503674859 +0000 UTC m=+103.604138679" watchObservedRunningTime="2026-09-17 17:58:43.512560588 +0000 UTC m=+105.613024408"
Sep 17 17:58:43.513115 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:43.513088    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jhhsj" podStartSLOduration=68.07579774 podStartE2EDuration="1m9.513077674s" totalImagesPullingTime="1.437279934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:43.512541189 +0000 UTC m=+105.613004999" watchObservedRunningTime="2026-09-17 17:58:43.513077674 +0000 UTC m=+105.613541486"
Sep 17 17:58:46.612393 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:46.612360    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-595f6ff97f-8xv29"
Sep 17 17:58:46.612829 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:46.612427    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-595f6ff97f-8xv29"
Sep 17 17:58:46.617140 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:46.617117    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-595f6ff97f-8xv29"
Sep 17 17:58:47.502253 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:47.502226    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-595f6ff97f-8xv29"
Sep 17 17:58:49.752359 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.752316    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-jxspn"]
Sep 17 17:58:49.755484 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.755465    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-jxspn"
Sep 17 17:58:49.760513 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.760489    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2mr86"]
Sep 17 17:58:49.761048 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.761028    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 17:58:49.762938 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.762918    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2mr86"
Sep 17 17:58:49.764429 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.764409    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 17:58:49.764682 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.764666    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 17:58:49.765473 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.765454    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 17:58:49.765579 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.765487    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-p7kbj\""
Sep 17 17:58:49.765579 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.765505    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 17:58:49.765685 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.765507    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 17:58:49.765765 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.765753    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 17:58:49.766457 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.766442    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 17:58:49.773432 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.773415    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 17:58:49.773724 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.773709    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vwrpp\""
Sep 17 17:58:49.789624 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.789598    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-jxspn"]
Sep 17 17:58:49.837859 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.837839    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ebeafe22-7486-457b-95cc-d9fd0d4496cf-node-exporter-tls\") pod \"node-exporter-2mr86\" (UID: \"ebeafe22-7486-457b-95cc-d9fd0d4496cf\") " pod="openshift-monitoring/node-exporter-2mr86"
Sep 17 17:58:49.837990 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.837874    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ebeafe22-7486-457b-95cc-d9fd0d4496cf-node-exporter-wtmp\") pod \"node-exporter-2mr86\" (UID: \"ebeafe22-7486-457b-95cc-d9fd0d4496cf\") " pod="openshift-monitoring/node-exporter-2mr86"
Sep 17 17:58:49.837990 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.837900    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebeafe22-7486-457b-95cc-d9fd0d4496cf-metrics-client-ca\") pod \"node-exporter-2mr86\" (UID: \"ebeafe22-7486-457b-95cc-d9fd0d4496cf\") " pod="openshift-monitoring/node-exporter-2mr86"
Sep 17 17:58:49.837990 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.837950    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b694aae6-aaf3-4091-9466-e336206bfc7e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-jxspn\" (UID: \"b694aae6-aaf3-4091-9466-e336206bfc7e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jxspn"
Sep 17 17:58:49.837990 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.837976    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ebeafe22-7486-457b-95cc-d9fd0d4496cf-node-exporter-textfile\") pod \"node-exporter-2mr86\" (UID: \"ebeafe22-7486-457b-95cc-d9fd0d4496cf\") " pod="openshift-monitoring/node-exporter-2mr86"
Sep 17 17:58:49.838257 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.838038    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9bbp\" (UniqueName: \"kubernetes.io/projected/b694aae6-aaf3-4091-9466-e336206bfc7e-kube-api-access-j9bbp\") pod \"kube-state-metrics-84c885658c-jxspn\" (UID: \"b694aae6-aaf3-4091-9466-e336206bfc7e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jxspn"
Sep 17 17:58:49.838257 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.838077    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4vlk\" (UniqueName: \"kubernetes.io/projected/ebeafe22-7486-457b-95cc-d9fd0d4496cf-kube-api-access-v4vlk\") pod \"node-exporter-2mr86\" (UID: \"ebeafe22-7486-457b-95cc-d9fd0d4496cf\") " pod="openshift-monitoring/node-exporter-2mr86"
Sep 17 17:58:49.838257 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.838113    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b694aae6-aaf3-4091-9466-e336206bfc7e-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-jxspn\" (UID: \"b694aae6-aaf3-4091-9466-e336206bfc7e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jxspn"
Sep 17 17:58:49.838257 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.838140    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebeafe22-7486-457b-95cc-d9fd0d4496cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2mr86\" (UID: \"ebeafe22-7486-457b-95cc-d9fd0d4496cf\") " pod="openshift-monitoring/node-exporter-2mr86"
Sep 17 17:58:49.838257 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.838182    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ebeafe22-7486-457b-95cc-d9fd0d4496cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-2mr86\" (UID: \"ebeafe22-7486-457b-95cc-d9fd0d4496cf\") " pod="openshift-monitoring/node-exporter-2mr86"
Sep 17 17:58:49.838257 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.838248    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ebeafe22-7486-457b-95cc-d9fd0d4496cf-sys\") pod \"node-exporter-2mr86\" (UID: \"ebeafe22-7486-457b-95cc-d9fd0d4496cf\") " pod="openshift-monitoring/node-exporter-2mr86"
Sep 17 17:58:49.838470 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.838291    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b694aae6-aaf3-4091-9466-e336206bfc7e-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-jxspn\" (UID: \"b694aae6-aaf3-4091-9466-e336206bfc7e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jxspn"
Sep 17 17:58:49.838470 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.838313    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b694aae6-aaf3-4091-9466-e336206bfc7e-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-jxspn\" (UID: \"b694aae6-aaf3-4091-9466-e336206bfc7e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jxspn"
Sep 17 17:58:49.838470 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.838346    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b694aae6-aaf3-4091-9466-e336206bfc7e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-jxspn\" (UID: \"b694aae6-aaf3-4091-9466-e336206bfc7e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jxspn"
Sep 17 17:58:49.838470 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.838367    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ebeafe22-7486-457b-95cc-d9fd0d4496cf-root\") pod \"node-exporter-2mr86\" (UID: \"ebeafe22-7486-457b-95cc-d9fd0d4496cf\") " pod="openshift-monitoring/node-exporter-2mr86"
Sep 17 17:58:49.939534 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.939502    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ebeafe22-7486-457b-95cc-d9fd0d4496cf-node-exporter-tls\") pod \"node-exporter-2mr86\" (UID: \"ebeafe22-7486-457b-95cc-d9fd0d4496cf\") " pod="openshift-monitoring/node-exporter-2mr86"
Sep 17 17:58:49.939534 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.939536    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ebeafe22-7486-457b-95cc-d9fd0d4496cf-node-exporter-wtmp\") pod \"node-exporter-2mr86\" (UID: \"ebeafe22-7486-457b-95cc-d9fd0d4496cf\") " pod="openshift-monitoring/node-exporter-2mr86"
Sep 17 17:58:49.939740 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.939555    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebeafe22-7486-457b-95cc-d9fd0d4496cf-metrics-client-ca\") pod \"node-exporter-2mr86\" (UID: \"ebeafe22-7486-457b-95cc-d9fd0d4496cf\") " pod="openshift-monitoring/node-exporter-2mr86"
Sep 17 17:58:49.939740 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.939587    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b694aae6-aaf3-4091-9466-e336206bfc7e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-jxspn\" (UID: \"b694aae6-aaf3-4091-9466-e336206bfc7e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jxspn"
Sep 17 17:58:49.939740 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.939611    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ebeafe22-7486-457b-95cc-d9fd0d4496cf-node-exporter-textfile\") pod \"node-exporter-2mr86\" (UID: \"ebeafe22-7486-457b-95cc-d9fd0d4496cf\") " pod="openshift-monitoring/node-exporter-2mr86"
Sep 17 17:58:49.939740 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.939652    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9bbp\" (UniqueName: \"kubernetes.io/projected/b694aae6-aaf3-4091-9466-e336206bfc7e-kube-api-access-j9bbp\") pod \"kube-state-metrics-84c885658c-jxspn\" (UID: \"b694aae6-aaf3-4091-9466-e336206bfc7e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jxspn"
Sep 17 17:58:49.939740 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.939679    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4vlk\" (UniqueName: \"kubernetes.io/projected/ebeafe22-7486-457b-95cc-d9fd0d4496cf-kube-api-access-v4vlk\") pod \"node-exporter-2mr86\" (UID: \"ebeafe22-7486-457b-95cc-d9fd0d4496cf\") " pod="openshift-monitoring/node-exporter-2mr86"
Sep 17 17:58:49.939740 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.939709    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b694aae6-aaf3-4091-9466-e336206bfc7e-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-jxspn\" (UID: \"b694aae6-aaf3-4091-9466-e336206bfc7e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jxspn"
Sep 17 17:58:49.939740 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.939713    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ebeafe22-7486-457b-95cc-d9fd0d4496cf-node-exporter-wtmp\") pod \"node-exporter-2mr86\" (UID: \"ebeafe22-7486-457b-95cc-d9fd0d4496cf\") " pod="openshift-monitoring/node-exporter-2mr86"
Sep 17 17:58:49.939740 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.939733    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebeafe22-7486-457b-95cc-d9fd0d4496cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2mr86\" (UID: \"ebeafe22-7486-457b-95cc-d9fd0d4496cf\") " pod="openshift-monitoring/node-exporter-2mr86"
Sep 17 17:58:49.940186 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.940081    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ebeafe22-7486-457b-95cc-d9fd0d4496cf-node-exporter-textfile\") pod \"node-exporter-2mr86\" (UID: \"ebeafe22-7486-457b-95cc-d9fd0d4496cf\") " pod="openshift-monitoring/node-exporter-2mr86"
Sep 17 17:58:49.940186 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.940143    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ebeafe22-7486-457b-95cc-d9fd0d4496cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-2mr86\" (UID: \"ebeafe22-7486-457b-95cc-d9fd0d4496cf\") " pod="openshift-monitoring/node-exporter-2mr86"
Sep 17 17:58:49.940312 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.940234    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ebeafe22-7486-457b-95cc-d9fd0d4496cf-sys\") pod \"node-exporter-2mr86\" (UID: \"ebeafe22-7486-457b-95cc-d9fd0d4496cf\") " pod="openshift-monitoring/node-exporter-2mr86"
Sep 17 17:58:49.940312 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.940246    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebeafe22-7486-457b-95cc-d9fd0d4496cf-metrics-client-ca\") pod \"node-exporter-2mr86\" (UID: \"ebeafe22-7486-457b-95cc-d9fd0d4496cf\") " pod="openshift-monitoring/node-exporter-2mr86"
Sep 17 17:58:49.940312 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.940303    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ebeafe22-7486-457b-95cc-d9fd0d4496cf-sys\") pod \"node-exporter-2mr86\" (UID: \"ebeafe22-7486-457b-95cc-d9fd0d4496cf\") " pod="openshift-monitoring/node-exporter-2mr86"
Sep 17 17:58:49.940312 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.940305    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b694aae6-aaf3-4091-9466-e336206bfc7e-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-jxspn\" (UID: \"b694aae6-aaf3-4091-9466-e336206bfc7e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jxspn"
Sep 17 17:58:49.940503 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.940334    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b694aae6-aaf3-4091-9466-e336206bfc7e-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-jxspn\" (UID: \"b694aae6-aaf3-4091-9466-e336206bfc7e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jxspn"
Sep 17 17:58:49.940503 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.940373    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b694aae6-aaf3-4091-9466-e336206bfc7e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-jxspn\" (UID: \"b694aae6-aaf3-4091-9466-e336206bfc7e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jxspn"
Sep 17 17:58:49.940503 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.940398    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ebeafe22-7486-457b-95cc-d9fd0d4496cf-root\") pod \"node-exporter-2mr86\" (UID: \"ebeafe22-7486-457b-95cc-d9fd0d4496cf\") " pod="openshift-monitoring/node-exporter-2mr86"
Sep 17 17:58:49.940503 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.940412    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b694aae6-aaf3-4091-9466-e336206bfc7e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-jxspn\" (UID: \"b694aae6-aaf3-4091-9466-e336206bfc7e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jxspn"
Sep 17 17:58:49.940503 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.940433    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b694aae6-aaf3-4091-9466-e336206bfc7e-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-jxspn\" (UID: \"b694aae6-aaf3-4091-9466-e336206bfc7e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jxspn"
Sep 17 17:58:49.940738 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.940575    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ebeafe22-7486-457b-95cc-d9fd0d4496cf-root\") pod \"node-exporter-2mr86\" (UID: \"ebeafe22-7486-457b-95cc-d9fd0d4496cf\") " pod="openshift-monitoring/node-exporter-2mr86"
Sep 17 17:58:49.940738 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.940669    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b694aae6-aaf3-4091-9466-e336206bfc7e-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-jxspn\" (UID: \"b694aae6-aaf3-4091-9466-e336206bfc7e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jxspn"
Sep 17 17:58:49.941020 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.940935    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ebeafe22-7486-457b-95cc-d9fd0d4496cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-2mr86\" (UID: \"ebeafe22-7486-457b-95cc-d9fd0d4496cf\") " pod="openshift-monitoring/node-exporter-2mr86"
Sep 17 17:58:49.942441 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.942413    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ebeafe22-7486-457b-95cc-d9fd0d4496cf-node-exporter-tls\") pod \"node-exporter-2mr86\" (UID: \"ebeafe22-7486-457b-95cc-d9fd0d4496cf\") " pod="openshift-monitoring/node-exporter-2mr86"
Sep 17 17:58:49.942883 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.942863    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b694aae6-aaf3-4091-9466-e336206bfc7e-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-jxspn\" (UID: \"b694aae6-aaf3-4091-9466-e336206bfc7e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jxspn"
Sep 17 17:58:49.943313 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.943292    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebeafe22-7486-457b-95cc-d9fd0d4496cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2mr86\" (UID: \"ebeafe22-7486-457b-95cc-d9fd0d4496cf\") " pod="openshift-monitoring/node-exporter-2mr86"
Sep 17 17:58:49.943452 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.943431    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b694aae6-aaf3-4091-9466-e336206bfc7e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-jxspn\" (UID: \"b694aae6-aaf3-4091-9466-e336206bfc7e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jxspn"
Sep 17 17:58:49.963665 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.963640    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4vlk\" (UniqueName: \"kubernetes.io/projected/ebeafe22-7486-457b-95cc-d9fd0d4496cf-kube-api-access-v4vlk\") pod \"node-exporter-2mr86\" (UID: \"ebeafe22-7486-457b-95cc-d9fd0d4496cf\") " pod="openshift-monitoring/node-exporter-2mr86"
Sep 17 17:58:49.964844 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:49.964819    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9bbp\" (UniqueName: \"kubernetes.io/projected/b694aae6-aaf3-4091-9466-e336206bfc7e-kube-api-access-j9bbp\") pod \"kube-state-metrics-84c885658c-jxspn\" (UID: \"b694aae6-aaf3-4091-9466-e336206bfc7e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jxspn"
Sep 17 17:58:50.067121 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:50.067061    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-jxspn"
Sep 17 17:58:50.074525 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:50.074503    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2mr86"
Sep 17 17:58:50.083742 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:58:50.083712    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebeafe22_7486_457b_95cc_d9fd0d4496cf.slice/crio-481ac70828ba3edf23e6148cdd30239dca3bc15df140dc2e1bfe9635bb2af8cc WatchSource:0}: Error finding container 481ac70828ba3edf23e6148cdd30239dca3bc15df140dc2e1bfe9635bb2af8cc: Status 404 returned error can't find the container with id 481ac70828ba3edf23e6148cdd30239dca3bc15df140dc2e1bfe9635bb2af8cc
Sep 17 17:58:50.398609 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:50.398582    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-jxspn"]
Sep 17 17:58:50.400403 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:58:50.400379    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb694aae6_aaf3_4091_9466_e336206bfc7e.slice/crio-dccd078625b88dfb361a726e9eeb71c32b9b1fd04bfa6477734a79ce4a149fb6 WatchSource:0}: Error finding container dccd078625b88dfb361a726e9eeb71c32b9b1fd04bfa6477734a79ce4a149fb6: Status 404 returned error can't find the container with id dccd078625b88dfb361a726e9eeb71c32b9b1fd04bfa6477734a79ce4a149fb6
Sep 17 17:58:50.511317 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:50.511281    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2mr86" event={"ID":"ebeafe22-7486-457b-95cc-d9fd0d4496cf","Type":"ContainerStarted","Data":"481ac70828ba3edf23e6148cdd30239dca3bc15df140dc2e1bfe9635bb2af8cc"}
Sep 17 17:58:50.512440 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:50.512418    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-jxspn" event={"ID":"b694aae6-aaf3-4091-9466-e336206bfc7e","Type":"ContainerStarted","Data":"dccd078625b88dfb361a726e9eeb71c32b9b1fd04bfa6477734a79ce4a149fb6"}
Sep 17 17:58:51.443704 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:51.443673    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56d8969c4d-qgjx4"
Sep 17 17:58:51.518523 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:51.518487    2581 generic.go:356] "Generic (PLEG): container finished" podID="ebeafe22-7486-457b-95cc-d9fd0d4496cf" containerID="874e2e108423291b78da1071b70c7b484cb1277c392902dd5b274a7d78d8bf98" exitCode=0
Sep 17 17:58:51.518700 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:51.518557    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2mr86" event={"ID":"ebeafe22-7486-457b-95cc-d9fd0d4496cf","Type":"ContainerDied","Data":"874e2e108423291b78da1071b70c7b484cb1277c392902dd5b274a7d78d8bf98"}
Sep 17 17:58:52.524921 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:52.524879    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2mr86" event={"ID":"ebeafe22-7486-457b-95cc-d9fd0d4496cf","Type":"ContainerStarted","Data":"706d3193e1b99d1635452d12fb7d0ad388cca0b09dc5f53b6441fe04fee65ee7"}
Sep 17 17:58:52.525401 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:52.524957    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2mr86" event={"ID":"ebeafe22-7486-457b-95cc-d9fd0d4496cf","Type":"ContainerStarted","Data":"2624992a77e2b933cebd41ca1bf46427021659afe1f4a61d6e83710fd33e510a"}
Sep 17 17:58:52.527189 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:52.527144    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-jxspn" event={"ID":"b694aae6-aaf3-4091-9466-e336206bfc7e","Type":"ContainerStarted","Data":"4e7bbe47c4e5b4c4ff3a0ea1d82305ecc2eac7add1bbd563f832892c53aeafc2"}
Sep 17 17:58:52.527298 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:52.527208    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-jxspn" event={"ID":"b694aae6-aaf3-4091-9466-e336206bfc7e","Type":"ContainerStarted","Data":"d6c84f80113fc49319dca09f3b5f2a344ada65e48ac5a21f4c897a9a54f3b0ab"}
Sep 17 17:58:52.527298 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:52.527228    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-jxspn" event={"ID":"b694aae6-aaf3-4091-9466-e336206bfc7e","Type":"ContainerStarted","Data":"7b685bbcdcb4ae759e14b6ebd76cf884169b13c682cc7bf784d99bb71907e593"}
Sep 17 17:58:52.551883 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:52.551840    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2mr86" podStartSLOduration=2.501910385 podStartE2EDuration="3.551825988s" totalImagesPullingTime="973.570509ms" totalInitContainerRuntime="76.345094ms" isStatefulPod=true podCreationTimestamp="2026-09-17 17:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:52.551373675 +0000 UTC m=+114.651837494" watchObservedRunningTime="2026-09-17 17:58:52.551825988 +0000 UTC m=+114.652289803"
Sep 17 17:58:52.580115 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:52.580072    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-jxspn" podStartSLOduration=2.212253432 podStartE2EDuration="3.580053745s" totalImagesPullingTime="1.367800313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:52.578239199 +0000 UTC m=+114.678703018" watchObservedRunningTime="2026-09-17 17:58:52.580053745 +0000 UTC m=+114.680517566"
Sep 17 17:58:53.491552 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:53.491525    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jhhsj"
Sep 17 17:58:53.654264 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:53.654227    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6787978d5c-jv2ww" podUID="d396cd04-2841-4a84-8fd3-235da3979f1c" containerName="registry" containerID="cri-o://4fbfa7904387c10692acdafbffc0a7e13a139ba62ee8498d0bc91598ff7703d6" gracePeriod=30
Sep 17 17:58:53.886936 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:53.886910    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:58:53.977978 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:53.977957    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d396cd04-2841-4a84-8fd3-235da3979f1c-trusted-ca\") pod \"d396cd04-2841-4a84-8fd3-235da3979f1c\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") "
Sep 17 17:58:53.978103 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:53.978004    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d396cd04-2841-4a84-8fd3-235da3979f1c-image-registry-private-configuration\") pod \"d396cd04-2841-4a84-8fd3-235da3979f1c\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") "
Sep 17 17:58:53.978103 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:53.978037    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d396cd04-2841-4a84-8fd3-235da3979f1c-registry-certificates\") pod \"d396cd04-2841-4a84-8fd3-235da3979f1c\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") "
Sep 17 17:58:53.978103 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:53.978056    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-registry-tls\") pod \"d396cd04-2841-4a84-8fd3-235da3979f1c\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") "
Sep 17 17:58:53.978103 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:53.978090    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-bound-sa-token\") pod \"d396cd04-2841-4a84-8fd3-235da3979f1c\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") "
Sep 17 17:58:53.978283 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:53.978106    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7mcf\" (UniqueName: \"kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-kube-api-access-r7mcf\") pod \"d396cd04-2841-4a84-8fd3-235da3979f1c\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") "
Sep 17 17:58:53.978283 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:53.978188    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d396cd04-2841-4a84-8fd3-235da3979f1c-ca-trust-extracted\") pod \"d396cd04-2841-4a84-8fd3-235da3979f1c\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") "
Sep 17 17:58:53.978283 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:53.978211    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d396cd04-2841-4a84-8fd3-235da3979f1c-installation-pull-secrets\") pod \"d396cd04-2841-4a84-8fd3-235da3979f1c\" (UID: \"d396cd04-2841-4a84-8fd3-235da3979f1c\") "
Sep 17 17:58:53.978467 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:53.978439    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d396cd04-2841-4a84-8fd3-235da3979f1c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d396cd04-2841-4a84-8fd3-235da3979f1c" (UID: "d396cd04-2841-4a84-8fd3-235da3979f1c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:58:53.978587 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:53.978562    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d396cd04-2841-4a84-8fd3-235da3979f1c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d396cd04-2841-4a84-8fd3-235da3979f1c" (UID: "d396cd04-2841-4a84-8fd3-235da3979f1c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:58:53.980806 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:53.980776    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d396cd04-2841-4a84-8fd3-235da3979f1c" (UID: "d396cd04-2841-4a84-8fd3-235da3979f1c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:58:53.980989 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:53.980968    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d396cd04-2841-4a84-8fd3-235da3979f1c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d396cd04-2841-4a84-8fd3-235da3979f1c" (UID: "d396cd04-2841-4a84-8fd3-235da3979f1c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:58:53.981060 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:53.981038    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-kube-api-access-r7mcf" (OuterVolumeSpecName: "kube-api-access-r7mcf") pod "d396cd04-2841-4a84-8fd3-235da3979f1c" (UID: "d396cd04-2841-4a84-8fd3-235da3979f1c"). InnerVolumeSpecName "kube-api-access-r7mcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:58:53.981060 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:53.981044    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d396cd04-2841-4a84-8fd3-235da3979f1c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d396cd04-2841-4a84-8fd3-235da3979f1c" (UID: "d396cd04-2841-4a84-8fd3-235da3979f1c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:58:53.981123 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:53.981036    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d396cd04-2841-4a84-8fd3-235da3979f1c" (UID: "d396cd04-2841-4a84-8fd3-235da3979f1c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:58:53.985825 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:53.985803    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d396cd04-2841-4a84-8fd3-235da3979f1c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d396cd04-2841-4a84-8fd3-235da3979f1c" (UID: "d396cd04-2841-4a84-8fd3-235da3979f1c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:58:54.079067 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.079041    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d396cd04-2841-4a84-8fd3-235da3979f1c-ca-trust-extracted\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 17:58:54.079067 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.079062    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d396cd04-2841-4a84-8fd3-235da3979f1c-installation-pull-secrets\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 17:58:54.079209 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.079072    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d396cd04-2841-4a84-8fd3-235da3979f1c-trusted-ca\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 17:58:54.079209 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.079081    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d396cd04-2841-4a84-8fd3-235da3979f1c-image-registry-private-configuration\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 17:58:54.079209 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.079089    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d396cd04-2841-4a84-8fd3-235da3979f1c-registry-certificates\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 17:58:54.079209 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.079097    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-registry-tls\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 17:58:54.079209 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.079105    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-bound-sa-token\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 17:58:54.079209 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.079112    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7mcf\" (UniqueName: \"kubernetes.io/projected/d396cd04-2841-4a84-8fd3-235da3979f1c-kube-api-access-r7mcf\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 17:58:54.350879 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.350809    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7644695647-fhdhz"]
Sep 17 17:58:54.351102 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.351089    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d396cd04-2841-4a84-8fd3-235da3979f1c" containerName="registry"
Sep 17 17:58:54.351145 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.351103    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d396cd04-2841-4a84-8fd3-235da3979f1c" containerName="registry"
Sep 17 17:58:54.351213 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.351177    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d396cd04-2841-4a84-8fd3-235da3979f1c" containerName="registry"
Sep 17 17:58:54.355684 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.355667    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7644695647-fhdhz"
Sep 17 17:58:54.358267 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.358234    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4xqls\""
Sep 17 17:58:54.358267 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.358253    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 17 17:58:54.358267 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.358235    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-d79h3g7nkhc2i\""
Sep 17 17:58:54.358478 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.358464    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 17:58:54.358532 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.358518    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 17 17:58:54.358532 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.358529    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 17 17:58:54.366702 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.366680    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7644695647-fhdhz"]
Sep 17 17:58:54.380688 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.380664    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5a2f9d6f-5315-40e6-9339-356a71997283-metrics-server-audit-profiles\") pod \"metrics-server-7644695647-fhdhz\" (UID: \"5a2f9d6f-5315-40e6-9339-356a71997283\") " pod="openshift-monitoring/metrics-server-7644695647-fhdhz"
Sep 17 17:58:54.380770 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.380714    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5a2f9d6f-5315-40e6-9339-356a71997283-audit-log\") pod \"metrics-server-7644695647-fhdhz\" (UID: \"5a2f9d6f-5315-40e6-9339-356a71997283\") " pod="openshift-monitoring/metrics-server-7644695647-fhdhz"
Sep 17 17:58:54.380770 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.380740    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5a2f9d6f-5315-40e6-9339-356a71997283-secret-metrics-server-client-certs\") pod \"metrics-server-7644695647-fhdhz\" (UID: \"5a2f9d6f-5315-40e6-9339-356a71997283\") " pod="openshift-monitoring/metrics-server-7644695647-fhdhz"
Sep 17 17:58:54.380883 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.380866    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znw5b\" (UniqueName: \"kubernetes.io/projected/5a2f9d6f-5315-40e6-9339-356a71997283-kube-api-access-znw5b\") pod \"metrics-server-7644695647-fhdhz\" (UID: \"5a2f9d6f-5315-40e6-9339-356a71997283\") " pod="openshift-monitoring/metrics-server-7644695647-fhdhz"
Sep 17 17:58:54.380933 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.380901    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a2f9d6f-5315-40e6-9339-356a71997283-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7644695647-fhdhz\" (UID: \"5a2f9d6f-5315-40e6-9339-356a71997283\") " pod="openshift-monitoring/metrics-server-7644695647-fhdhz"
Sep 17 17:58:54.380933 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.380928    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5a2f9d6f-5315-40e6-9339-356a71997283-secret-metrics-server-tls\") pod \"metrics-server-7644695647-fhdhz\" (UID: \"5a2f9d6f-5315-40e6-9339-356a71997283\") " pod="openshift-monitoring/metrics-server-7644695647-fhdhz"
Sep 17 17:58:54.381031 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.380993    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5a2f9d6f-5315-40e6-9339-356a71997283-client-ca-bundle\") pod \"metrics-server-7644695647-fhdhz\" (UID: \"5a2f9d6f-5315-40e6-9339-356a71997283\") " pod="openshift-monitoring/metrics-server-7644695647-fhdhz"
Sep 17 17:58:54.481682 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.481659    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5a2f9d6f-5315-40e6-9339-356a71997283-metrics-server-audit-profiles\") pod \"metrics-server-7644695647-fhdhz\" (UID: \"5a2f9d6f-5315-40e6-9339-356a71997283\") " pod="openshift-monitoring/metrics-server-7644695647-fhdhz"
Sep 17 17:58:54.481758 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.481695    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5a2f9d6f-5315-40e6-9339-356a71997283-audit-log\") pod \"metrics-server-7644695647-fhdhz\" (UID: \"5a2f9d6f-5315-40e6-9339-356a71997283\") " pod="openshift-monitoring/metrics-server-7644695647-fhdhz"
Sep 17 17:58:54.481758 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.481718    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5a2f9d6f-5315-40e6-9339-356a71997283-secret-metrics-server-client-certs\") pod \"metrics-server-7644695647-fhdhz\" (UID: \"5a2f9d6f-5315-40e6-9339-356a71997283\") " pod="openshift-monitoring/metrics-server-7644695647-fhdhz"
Sep 17 17:58:54.481758 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.481751    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znw5b\" (UniqueName: \"kubernetes.io/projected/5a2f9d6f-5315-40e6-9339-356a71997283-kube-api-access-znw5b\") pod \"metrics-server-7644695647-fhdhz\" (UID: \"5a2f9d6f-5315-40e6-9339-356a71997283\") " pod="openshift-monitoring/metrics-server-7644695647-fhdhz"
Sep 17 17:58:54.481878 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.481778    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a2f9d6f-5315-40e6-9339-356a71997283-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7644695647-fhdhz\" (UID: \"5a2f9d6f-5315-40e6-9339-356a71997283\") " pod="openshift-monitoring/metrics-server-7644695647-fhdhz"
Sep 17 17:58:54.481878 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.481802    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5a2f9d6f-5315-40e6-9339-356a71997283-secret-metrics-server-tls\") pod \"metrics-server-7644695647-fhdhz\" (UID: \"5a2f9d6f-5315-40e6-9339-356a71997283\") " pod="openshift-monitoring/metrics-server-7644695647-fhdhz"
Sep 17 17:58:54.481878 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.481834    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5a2f9d6f-5315-40e6-9339-356a71997283-client-ca-bundle\") pod \"metrics-server-7644695647-fhdhz\" (UID: \"5a2f9d6f-5315-40e6-9339-356a71997283\") " pod="openshift-monitoring/metrics-server-7644695647-fhdhz"
Sep 17 17:58:54.482115 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.482095    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5a2f9d6f-5315-40e6-9339-356a71997283-audit-log\") pod \"metrics-server-7644695647-fhdhz\" (UID: \"5a2f9d6f-5315-40e6-9339-356a71997283\") " pod="openshift-monitoring/metrics-server-7644695647-fhdhz"
Sep 17 17:58:54.482681 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.482663    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5a2f9d6f-5315-40e6-9339-356a71997283-metrics-server-audit-profiles\") pod \"metrics-server-7644695647-fhdhz\" (UID: \"5a2f9d6f-5315-40e6-9339-356a71997283\") " pod="openshift-monitoring/metrics-server-7644695647-fhdhz"
Sep 17 17:58:54.482769 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.482648    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a2f9d6f-5315-40e6-9339-356a71997283-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7644695647-fhdhz\" (UID: \"5a2f9d6f-5315-40e6-9339-356a71997283\") " pod="openshift-monitoring/metrics-server-7644695647-fhdhz"
Sep 17 17:58:54.484276 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.484259    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5a2f9d6f-5315-40e6-9339-356a71997283-client-ca-bundle\") pod \"metrics-server-7644695647-fhdhz\" (UID: \"5a2f9d6f-5315-40e6-9339-356a71997283\") " pod="openshift-monitoring/metrics-server-7644695647-fhdhz"
Sep 17 17:58:54.484361 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.484330    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5a2f9d6f-5315-40e6-9339-356a71997283-secret-metrics-server-client-certs\") pod \"metrics-server-7644695647-fhdhz\" (UID: \"5a2f9d6f-5315-40e6-9339-356a71997283\") " pod="openshift-monitoring/metrics-server-7644695647-fhdhz"
Sep 17 17:58:54.484411 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.484394    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5a2f9d6f-5315-40e6-9339-356a71997283-secret-metrics-server-tls\") pod \"metrics-server-7644695647-fhdhz\" (UID: \"5a2f9d6f-5315-40e6-9339-356a71997283\") " pod="openshift-monitoring/metrics-server-7644695647-fhdhz"
Sep 17 17:58:54.491024 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.491002    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znw5b\" (UniqueName: \"kubernetes.io/projected/5a2f9d6f-5315-40e6-9339-356a71997283-kube-api-access-znw5b\") pod \"metrics-server-7644695647-fhdhz\" (UID: \"5a2f9d6f-5315-40e6-9339-356a71997283\") " pod="openshift-monitoring/metrics-server-7644695647-fhdhz"
Sep 17 17:58:54.535920 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.535896    2581 generic.go:356] "Generic (PLEG): container finished" podID="d396cd04-2841-4a84-8fd3-235da3979f1c" containerID="4fbfa7904387c10692acdafbffc0a7e13a139ba62ee8498d0bc91598ff7703d6" exitCode=0
Sep 17 17:58:54.536013 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.535927    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6787978d5c-jv2ww" event={"ID":"d396cd04-2841-4a84-8fd3-235da3979f1c","Type":"ContainerDied","Data":"4fbfa7904387c10692acdafbffc0a7e13a139ba62ee8498d0bc91598ff7703d6"}
Sep 17 17:58:54.536013 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.535936    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6787978d5c-jv2ww"
Sep 17 17:58:54.536013 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.535963    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6787978d5c-jv2ww" event={"ID":"d396cd04-2841-4a84-8fd3-235da3979f1c","Type":"ContainerDied","Data":"98e0f0d736e30e6ab945ada946877c9ecd1f3b920af88cb49a939b3d765bde9b"}
Sep 17 17:58:54.536013 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.535991    2581 scope.go:117] "RemoveContainer" containerID="4fbfa7904387c10692acdafbffc0a7e13a139ba62ee8498d0bc91598ff7703d6"
Sep 17 17:58:54.543543 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.543526    2581 scope.go:117] "RemoveContainer" containerID="4fbfa7904387c10692acdafbffc0a7e13a139ba62ee8498d0bc91598ff7703d6"
Sep 17 17:58:54.543829 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:58:54.543806    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fbfa7904387c10692acdafbffc0a7e13a139ba62ee8498d0bc91598ff7703d6\": container with ID starting with 4fbfa7904387c10692acdafbffc0a7e13a139ba62ee8498d0bc91598ff7703d6 not found: ID does not exist" containerID="4fbfa7904387c10692acdafbffc0a7e13a139ba62ee8498d0bc91598ff7703d6"
Sep 17 17:58:54.543872 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.543828    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fbfa7904387c10692acdafbffc0a7e13a139ba62ee8498d0bc91598ff7703d6"} err="failed to get container status \"4fbfa7904387c10692acdafbffc0a7e13a139ba62ee8498d0bc91598ff7703d6\": rpc error: code = NotFound desc = could not find container \"4fbfa7904387c10692acdafbffc0a7e13a139ba62ee8498d0bc91598ff7703d6\": container with ID starting with 4fbfa7904387c10692acdafbffc0a7e13a139ba62ee8498d0bc91598ff7703d6 not found: ID does not exist"
Sep 17 17:58:54.552186 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.552145    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6787978d5c-jv2ww"]
Sep 17 17:58:54.558604 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.558588    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6787978d5c-jv2ww"]
Sep 17 17:58:54.664693 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.664667    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7644695647-fhdhz"
Sep 17 17:58:54.952220 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.952114    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7644695647-fhdhz"]
Sep 17 17:58:54.956374 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:58:54.956343    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a2f9d6f_5315_40e6_9339_356a71997283.slice/crio-10e82813e6ef2339d145d7457f2f6cfaaca4d3e884624a46595fc2ee4f376678 WatchSource:0}: Error finding container 10e82813e6ef2339d145d7457f2f6cfaaca4d3e884624a46595fc2ee4f376678: Status 404 returned error can't find the container with id 10e82813e6ef2339d145d7457f2f6cfaaca4d3e884624a46595fc2ee4f376678
Sep 17 17:58:54.977997 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.977973    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws"]
Sep 17 17:58:54.982635 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.982609    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws"
Sep 17 17:58:54.986254 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.986232    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 17:58:54.986570 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.986554    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 17:58:54.987064 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.987043    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 17:58:54.987185 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.987094    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 17:58:54.987291 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.987270    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-nwf9s\""
Sep 17 17:58:54.990289 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.990269    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 17:58:54.993623 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:54.993606    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 17:58:55.013996 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:55.013972    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws"]
Sep 17 17:58:55.087089 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:55.087065    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85a6db98-a429-4f14-8d1e-98ed401382d9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b9cffb6c-cf2ws\" (UID: \"85a6db98-a429-4f14-8d1e-98ed401382d9\") " pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws"
Sep 17 17:58:55.087197 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:55.087108    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85a6db98-a429-4f14-8d1e-98ed401382d9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b9cffb6c-cf2ws\" (UID: \"85a6db98-a429-4f14-8d1e-98ed401382d9\") " pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws"
Sep 17 17:58:55.087197 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:55.087134    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/85a6db98-a429-4f14-8d1e-98ed401382d9-telemeter-client-tls\") pod \"telemeter-client-5b9cffb6c-cf2ws\" (UID: \"85a6db98-a429-4f14-8d1e-98ed401382d9\") " pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws"
Sep 17 17:58:55.087278 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:55.087210    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85a6db98-a429-4f14-8d1e-98ed401382d9-serving-certs-ca-bundle\") pod \"telemeter-client-5b9cffb6c-cf2ws\" (UID: \"85a6db98-a429-4f14-8d1e-98ed401382d9\") " pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws"
Sep 17 17:58:55.087278 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:55.087241    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85a6db98-a429-4f14-8d1e-98ed401382d9-metrics-client-ca\") pod \"telemeter-client-5b9cffb6c-cf2ws\" (UID: \"85a6db98-a429-4f14-8d1e-98ed401382d9\") " pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws"
Sep 17 17:58:55.087278 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:55.087266    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/85a6db98-a429-4f14-8d1e-98ed401382d9-secret-telemeter-client\") pod \"telemeter-client-5b9cffb6c-cf2ws\" (UID: \"85a6db98-a429-4f14-8d1e-98ed401382d9\") " pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws"
Sep 17 17:58:55.087369 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:55.087287    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/85a6db98-a429-4f14-8d1e-98ed401382d9-federate-client-tls\") pod \"telemeter-client-5b9cffb6c-cf2ws\" (UID: \"85a6db98-a429-4f14-8d1e-98ed401382d9\") " pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws"
Sep 17 17:58:55.087369 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:55.087304    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2n5d\" (UniqueName: \"kubernetes.io/projected/85a6db98-a429-4f14-8d1e-98ed401382d9-kube-api-access-v2n5d\") pod \"telemeter-client-5b9cffb6c-cf2ws\" (UID: \"85a6db98-a429-4f14-8d1e-98ed401382d9\") " pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws"
Sep 17 17:58:55.188498 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:55.188469    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85a6db98-a429-4f14-8d1e-98ed401382d9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b9cffb6c-cf2ws\" (UID: \"85a6db98-a429-4f14-8d1e-98ed401382d9\") " pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws"
Sep 17 17:58:55.188498 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:55.188499    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/85a6db98-a429-4f14-8d1e-98ed401382d9-telemeter-client-tls\") pod \"telemeter-client-5b9cffb6c-cf2ws\" (UID: \"85a6db98-a429-4f14-8d1e-98ed401382d9\") " pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws"
Sep 17 17:58:55.188669 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:55.188515    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85a6db98-a429-4f14-8d1e-98ed401382d9-serving-certs-ca-bundle\") pod \"telemeter-client-5b9cffb6c-cf2ws\" (UID: \"85a6db98-a429-4f14-8d1e-98ed401382d9\") " pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws"
Sep 17 17:58:55.188669 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:55.188533    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85a6db98-a429-4f14-8d1e-98ed401382d9-metrics-client-ca\") pod \"telemeter-client-5b9cffb6c-cf2ws\" (UID: \"85a6db98-a429-4f14-8d1e-98ed401382d9\") " pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws"
Sep 17 17:58:55.188669 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:55.188556    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/85a6db98-a429-4f14-8d1e-98ed401382d9-secret-telemeter-client\") pod \"telemeter-client-5b9cffb6c-cf2ws\" (UID: \"85a6db98-a429-4f14-8d1e-98ed401382d9\") " pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws"
Sep 17 17:58:55.188811 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:55.188797    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/85a6db98-a429-4f14-8d1e-98ed401382d9-federate-client-tls\") pod \"telemeter-client-5b9cffb6c-cf2ws\" (UID: \"85a6db98-a429-4f14-8d1e-98ed401382d9\") " pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws"
Sep 17 17:58:55.188874 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:55.188818    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2n5d\" (UniqueName: \"kubernetes.io/projected/85a6db98-a429-4f14-8d1e-98ed401382d9-kube-api-access-v2n5d\") pod \"telemeter-client-5b9cffb6c-cf2ws\" (UID: \"85a6db98-a429-4f14-8d1e-98ed401382d9\") " pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws"
Sep 17 17:58:55.188874 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:55.188859    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85a6db98-a429-4f14-8d1e-98ed401382d9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b9cffb6c-cf2ws\" (UID: \"85a6db98-a429-4f14-8d1e-98ed401382d9\") " pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws"
Sep 17 17:58:55.189585 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:55.189558    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85a6db98-a429-4f14-8d1e-98ed401382d9-serving-certs-ca-bundle\") pod \"telemeter-client-5b9cffb6c-cf2ws\" (UID: \"85a6db98-a429-4f14-8d1e-98ed401382d9\") " pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws"
Sep 17 17:58:55.189676 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:55.189572    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85a6db98-a429-4f14-8d1e-98ed401382d9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b9cffb6c-cf2ws\" (UID: \"85a6db98-a429-4f14-8d1e-98ed401382d9\") " pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws"
Sep 17 17:58:55.189676 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:55.189644    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85a6db98-a429-4f14-8d1e-98ed401382d9-metrics-client-ca\") pod \"telemeter-client-5b9cffb6c-cf2ws\" (UID: \"85a6db98-a429-4f14-8d1e-98ed401382d9\") " pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws"
Sep 17 17:58:55.191232 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:55.191207    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/85a6db98-a429-4f14-8d1e-98ed401382d9-telemeter-client-tls\") pod \"telemeter-client-5b9cffb6c-cf2ws\" (UID: \"85a6db98-a429-4f14-8d1e-98ed401382d9\") " pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws"
Sep 17 17:58:55.191301 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:55.191215    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/85a6db98-a429-4f14-8d1e-98ed401382d9-secret-telemeter-client\") pod \"telemeter-client-5b9cffb6c-cf2ws\" (UID: \"85a6db98-a429-4f14-8d1e-98ed401382d9\") " pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws"
Sep 17 17:58:55.191637 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:55.191621    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/85a6db98-a429-4f14-8d1e-98ed401382d9-federate-client-tls\") pod \"telemeter-client-5b9cffb6c-cf2ws\" (UID: \"85a6db98-a429-4f14-8d1e-98ed401382d9\") " pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws"
Sep 17 17:58:55.191701 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:55.191685    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85a6db98-a429-4f14-8d1e-98ed401382d9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b9cffb6c-cf2ws\" (UID: \"85a6db98-a429-4f14-8d1e-98ed401382d9\") " pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws"
Sep 17 17:58:55.197520 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:55.197498    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2n5d\" (UniqueName: \"kubernetes.io/projected/85a6db98-a429-4f14-8d1e-98ed401382d9-kube-api-access-v2n5d\") pod \"telemeter-client-5b9cffb6c-cf2ws\" (UID: \"85a6db98-a429-4f14-8d1e-98ed401382d9\") " pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws"
Sep 17 17:58:55.293577 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:55.293507    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws"
Sep 17 17:58:55.541562 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:55.541523    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7644695647-fhdhz" event={"ID":"5a2f9d6f-5315-40e6-9339-356a71997283","Type":"ContainerStarted","Data":"10e82813e6ef2339d145d7457f2f6cfaaca4d3e884624a46595fc2ee4f376678"}
Sep 17 17:58:55.583123 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:55.583004    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws"]
Sep 17 17:58:55.587309 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:58:55.587285    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85a6db98_a429_4f14_8d1e_98ed401382d9.slice/crio-655565f6f05a7379bf55a5c3f13dedba595369befd861a8f640e0a91589fa0ff WatchSource:0}: Error finding container 655565f6f05a7379bf55a5c3f13dedba595369befd861a8f640e0a91589fa0ff: Status 404 returned error can't find the container with id 655565f6f05a7379bf55a5c3f13dedba595369befd861a8f640e0a91589fa0ff
Sep 17 17:58:56.050560 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.050526    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d396cd04-2841-4a84-8fd3-235da3979f1c" path="/var/lib/kubelet/pods/d396cd04-2841-4a84-8fd3-235da3979f1c/volumes"
Sep 17 17:58:56.411328 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.411295    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:58:56.419257 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.419223    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.446511 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.446482    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 17:58:56.446665 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.446488    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 17:58:56.446753 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.446734    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 17:58:56.446841 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.446753    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 17:58:56.446841 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.446759    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 17:58:56.446936 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.446859    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 17:58:56.447003 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.446988    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 17:58:56.447136 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.447071    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-26otalvshcc79\""
Sep 17 17:58:56.447136 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.447109    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-s8jdl\""
Sep 17 17:58:56.447136 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.447124    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 17:58:56.447313 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.447263    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 17:58:56.447313 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.447308    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 17:58:56.447407 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.447338    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 17:58:56.447407 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.447366    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 17:58:56.447907 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.447690    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 17:58:56.447907 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.447757    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 17:58:56.500935 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.500907    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.501056 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.500944    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.501056 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.500967    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.501056 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.501001    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.501200 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.501062    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.501200 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.501098    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-config\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.501200 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.501121    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.501200 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.501168    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.501325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.501210    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqpsr\" (UniqueName: \"kubernetes.io/projected/744154b5-f9e0-45f5-a35f-f522ef3f15a4-kube-api-access-bqpsr\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.501325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.501227    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/744154b5-f9e0-45f5-a35f-f522ef3f15a4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.501325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.501244    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.501325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.501285    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/744154b5-f9e0-45f5-a35f-f522ef3f15a4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.501325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.501298    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.501325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.501313    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.501485 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.501339    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.501485 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.501361    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.501485 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.501379    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.501485 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.501395    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/744154b5-f9e0-45f5-a35f-f522ef3f15a4-config-out\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.501485 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.501410    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.501485 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.501441    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-web-config\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.548539 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.548500    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7644695647-fhdhz" event={"ID":"5a2f9d6f-5315-40e6-9339-356a71997283","Type":"ContainerStarted","Data":"0991460c6f462c66bf91d5a1a9a9b752114ac80be79899990502bee9fc2d2ba0"}
Sep 17 17:58:56.549997 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.549964    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws" event={"ID":"85a6db98-a429-4f14-8d1e-98ed401382d9","Type":"ContainerStarted","Data":"655565f6f05a7379bf55a5c3f13dedba595369befd861a8f640e0a91589fa0ff"}
Sep 17 17:58:56.602110 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.602077    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqpsr\" (UniqueName: \"kubernetes.io/projected/744154b5-f9e0-45f5-a35f-f522ef3f15a4-kube-api-access-bqpsr\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.602110 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.602112    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/744154b5-f9e0-45f5-a35f-f522ef3f15a4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.602348 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.602203    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.602348 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.602266    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/744154b5-f9e0-45f5-a35f-f522ef3f15a4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.602348 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.602295    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.602348 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.602322    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.602545 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.602354    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.602545 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.602380    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.602545 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.602410    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.602545 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.602451    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/744154b5-f9e0-45f5-a35f-f522ef3f15a4-config-out\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.602545 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.602475    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.602545 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.602499    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-web-config\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.602545 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.602547    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.602883 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.602565    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/744154b5-f9e0-45f5-a35f-f522ef3f15a4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.602883 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.602604    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.602883 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.602636    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.602883 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.602654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.602883 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.602701    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.602883 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.602745    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-config\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.602883 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.602770    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.602883 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.602795    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.602883 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.602842    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.603341 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.603129    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.604354 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.604302    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.604354 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.604347    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.605294 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.605269    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.606602 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.606434    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.607184 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.607091    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.607508 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.607444    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/744154b5-f9e0-45f5-a35f-f522ef3f15a4-config-out\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.607595 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.607528    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.607595 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.607582    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.607951 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.607835    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.608033 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.608006    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/744154b5-f9e0-45f5-a35f-f522ef3f15a4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.608100 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.608060    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.608352 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.608307    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.608946 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.608923    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.609093 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.609076    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-config\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.609793 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.609773    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.609890 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.609834    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-web-config\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.615413 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.615343    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:58:56.638752 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.638724    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqpsr\" (UniqueName: \"kubernetes.io/projected/744154b5-f9e0-45f5-a35f-f522ef3f15a4-kube-api-access-bqpsr\") pod \"prometheus-k8s-0\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:56.652261 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.652208    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7644695647-fhdhz" podStartSLOduration=1.468898931 podStartE2EDuration="2.652190513s" totalImagesPullingTime="1.183291582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:56.651527836 +0000 UTC m=+118.751991655" watchObservedRunningTime="2026-09-17 17:58:56.652190513 +0000 UTC m=+118.752654333"
Sep 17 17:58:56.730631 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:56.730568    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:58:57.035382 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:57.035297    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:58:57.039010 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:58:57.038982    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod744154b5_f9e0_45f5_a35f_f522ef3f15a4.slice/crio-fcab4654017dc3b72bfc4e1064df57727ad814aeec4a6a5d627177fb84459262 WatchSource:0}: Error finding container fcab4654017dc3b72bfc4e1064df57727ad814aeec4a6a5d627177fb84459262: Status 404 returned error can't find the container with id fcab4654017dc3b72bfc4e1064df57727ad814aeec4a6a5d627177fb84459262
Sep 17 17:58:57.554738 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:57.554704    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"744154b5-f9e0-45f5-a35f-f522ef3f15a4","Type":"ContainerStarted","Data":"fcab4654017dc3b72bfc4e1064df57727ad814aeec4a6a5d627177fb84459262"}
Sep 17 17:58:58.561099 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:58.561062    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws" event={"ID":"85a6db98-a429-4f14-8d1e-98ed401382d9","Type":"ContainerStarted","Data":"038848971c72a9a8627bf48f89b6360375422e241a65950641c9e1da2c2adece"}
Sep 17 17:58:58.561562 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:58.561123    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws" event={"ID":"85a6db98-a429-4f14-8d1e-98ed401382d9","Type":"ContainerStarted","Data":"4d093903162591999eadcdeb91a7e3d384dee65fe16e8f2c489d6070cb846c83"}
Sep 17 17:58:58.561562 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:58.561186    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws" event={"ID":"85a6db98-a429-4f14-8d1e-98ed401382d9","Type":"ContainerStarted","Data":"aa6bff0b17a6bf81d8e23250110904a9452d673759e5dc053c4dfc17aa7895e5"}
Sep 17 17:58:58.562674 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:58.562643    2581 generic.go:356] "Generic (PLEG): container finished" podID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerID="962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f" exitCode=0
Sep 17 17:58:58.562782 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:58.562726    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"744154b5-f9e0-45f5-a35f-f522ef3f15a4","Type":"ContainerDied","Data":"962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f"}
Sep 17 17:58:58.598971 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:58:58.598927    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5b9cffb6c-cf2ws" podStartSLOduration=2.135119645 podStartE2EDuration="4.598911743s" totalImagesPullingTime="2.463792098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:58.598857777 +0000 UTC m=+120.699321594" watchObservedRunningTime="2026-09-17 17:58:58.598911743 +0000 UTC m=+120.699375562"
Sep 17 17:59:01.576533 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:01.576501    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"744154b5-f9e0-45f5-a35f-f522ef3f15a4","Type":"ContainerStarted","Data":"5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837"}
Sep 17 17:59:01.576856 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:01.576572    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"744154b5-f9e0-45f5-a35f-f522ef3f15a4","Type":"ContainerStarted","Data":"ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace"}
Sep 17 17:59:03.588339 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:03.588258    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"744154b5-f9e0-45f5-a35f-f522ef3f15a4","Type":"ContainerStarted","Data":"20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05"}
Sep 17 17:59:03.588753 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:03.588361    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"744154b5-f9e0-45f5-a35f-f522ef3f15a4","Type":"ContainerStarted","Data":"5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6"}
Sep 17 17:59:03.588753 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:03.588408    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"744154b5-f9e0-45f5-a35f-f522ef3f15a4","Type":"ContainerStarted","Data":"8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935"}
Sep 17 17:59:03.588753 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:03.588448    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"744154b5-f9e0-45f5-a35f-f522ef3f15a4","Type":"ContainerStarted","Data":"f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891"}
Sep 17 17:59:03.672766 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:03.672711    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.073892827 podStartE2EDuration="7.672691361s" totalImagesPullingTime="5.579677239s" totalInitContainerRuntime="19.121295ms" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:03.671500419 +0000 UTC m=+125.771964230" watchObservedRunningTime="2026-09-17 17:59:03.672691361 +0000 UTC m=+125.773155180"
Sep 17 17:59:06.731020 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:06.730977    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:59:09.375371 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:09.375337    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-595f6ff97f-8xv29"]
Sep 17 17:59:09.730078 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:09.729997    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11b6807f-cb26-4117-b06d-eb974e5640ae-metrics-certs\") pod \"network-metrics-daemon-dsz4t\" (UID: \"11b6807f-cb26-4117-b06d-eb974e5640ae\") " pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:59:09.732354 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:09.732337    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11b6807f-cb26-4117-b06d-eb974e5640ae-metrics-certs\") pod \"network-metrics-daemon-dsz4t\" (UID: \"11b6807f-cb26-4117-b06d-eb974e5640ae\") " pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:59:09.863135 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:09.863109    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s7697\""
Sep 17 17:59:09.868220 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:09.868205    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsz4t"
Sep 17 17:59:10.161931 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:10.161902    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dsz4t"]
Sep 17 17:59:10.165074 ip-10-0-134-209 kubenswrapper[2581]: W0917 17:59:10.165039    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b6807f_cb26_4117_b06d_eb974e5640ae.slice/crio-7ba11c454952b14ec82696ed05e1afe7cc4ccbf102ecc3b56981a7d9cc3cedbc WatchSource:0}: Error finding container 7ba11c454952b14ec82696ed05e1afe7cc4ccbf102ecc3b56981a7d9cc3cedbc: Status 404 returned error can't find the container with id 7ba11c454952b14ec82696ed05e1afe7cc4ccbf102ecc3b56981a7d9cc3cedbc
Sep 17 17:59:10.619545 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:10.619507    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dsz4t" event={"ID":"11b6807f-cb26-4117-b06d-eb974e5640ae","Type":"ContainerStarted","Data":"7ba11c454952b14ec82696ed05e1afe7cc4ccbf102ecc3b56981a7d9cc3cedbc"}
Sep 17 17:59:11.630531 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:11.630489    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dsz4t" event={"ID":"11b6807f-cb26-4117-b06d-eb974e5640ae","Type":"ContainerStarted","Data":"fc1fe1e89841e9f8bb2f5bb4b260c641ae072bac6d115781a70151f8b20a9d25"}
Sep 17 17:59:11.630873 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:11.630585    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dsz4t" event={"ID":"11b6807f-cb26-4117-b06d-eb974e5640ae","Type":"ContainerStarted","Data":"2bb08d6f7e7fbef2c6dd8d7dd044aae7e41063a6d61d787c160a2da5d01582d1"}
Sep 17 17:59:11.650401 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:11.650352    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dsz4t" podStartSLOduration=130.57669378 podStartE2EDuration="2m11.650336715s" totalImagesPullingTime="1.073642935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:11.649285663 +0000 UTC m=+133.749749484" watchObservedRunningTime="2026-09-17 17:59:11.650336715 +0000 UTC m=+133.750800533"
Sep 17 17:59:14.664801 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:14.664770    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7644695647-fhdhz"
Sep 17 17:59:14.665285 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:14.664815    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7644695647-fhdhz"
Sep 17 17:59:16.651491 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:16.651456    2581 generic.go:356] "Generic (PLEG): container finished" podID="1080ecc5-cca9-47db-b949-05bdd3777351" containerID="0538f9bba6a5434158bd3c1f66b5ddf7507b9e4bfcf30414a27eaea89f21eea6" exitCode=0
Sep 17 17:59:16.651898 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:16.651532    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-ztc5j" event={"ID":"1080ecc5-cca9-47db-b949-05bdd3777351","Type":"ContainerDied","Data":"0538f9bba6a5434158bd3c1f66b5ddf7507b9e4bfcf30414a27eaea89f21eea6"}
Sep 17 17:59:16.651898 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:16.651831    2581 scope.go:117] "RemoveContainer" containerID="0538f9bba6a5434158bd3c1f66b5ddf7507b9e4bfcf30414a27eaea89f21eea6"
Sep 17 17:59:17.657777 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:17.657738    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-ztc5j" event={"ID":"1080ecc5-cca9-47db-b949-05bdd3777351","Type":"ContainerStarted","Data":"3fd35be60cda076d99b0a2ad4003f1f5776995c21d90bacb8a2b775411e28e2e"}
Sep 17 17:59:34.399671 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.399612    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-595f6ff97f-8xv29" podUID="01a3be75-f0e0-4691-8365-bf0b19b2aeb2" containerName="console" containerID="cri-o://38ff75e4bfd711cca31205f2faa1b2c15dc3c21fd93eb56c23ed16eb3ddf65f2" gracePeriod=15
Sep 17 17:59:34.670593 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.670537    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7644695647-fhdhz"
Sep 17 17:59:34.671089 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.671066    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595f6ff97f-8xv29"
Sep 17 17:59:34.674231 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.674209    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7644695647-fhdhz"
Sep 17 17:59:34.733685 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.733623    2581 generic.go:356] "Generic (PLEG): container finished" podID="01a3be75-f0e0-4691-8365-bf0b19b2aeb2" containerID="38ff75e4bfd711cca31205f2faa1b2c15dc3c21fd93eb56c23ed16eb3ddf65f2" exitCode=0
Sep 17 17:59:34.733685 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.733666    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595f6ff97f-8xv29"
Sep 17 17:59:34.733877 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.733734    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595f6ff97f-8xv29" event={"ID":"01a3be75-f0e0-4691-8365-bf0b19b2aeb2","Type":"ContainerDied","Data":"38ff75e4bfd711cca31205f2faa1b2c15dc3c21fd93eb56c23ed16eb3ddf65f2"}
Sep 17 17:59:34.733877 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.733807    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595f6ff97f-8xv29" event={"ID":"01a3be75-f0e0-4691-8365-bf0b19b2aeb2","Type":"ContainerDied","Data":"30858cabe95cea21928b78000d8982d27c05910514fe57e9836be8640be83e7f"}
Sep 17 17:59:34.733877 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.733869    2581 scope.go:117] "RemoveContainer" containerID="38ff75e4bfd711cca31205f2faa1b2c15dc3c21fd93eb56c23ed16eb3ddf65f2"
Sep 17 17:59:34.744384 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.744363    2581 scope.go:117] "RemoveContainer" containerID="38ff75e4bfd711cca31205f2faa1b2c15dc3c21fd93eb56c23ed16eb3ddf65f2"
Sep 17 17:59:34.744989 ip-10-0-134-209 kubenswrapper[2581]: E0917 17:59:34.744970    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38ff75e4bfd711cca31205f2faa1b2c15dc3c21fd93eb56c23ed16eb3ddf65f2\": container with ID starting with 38ff75e4bfd711cca31205f2faa1b2c15dc3c21fd93eb56c23ed16eb3ddf65f2 not found: ID does not exist" containerID="38ff75e4bfd711cca31205f2faa1b2c15dc3c21fd93eb56c23ed16eb3ddf65f2"
Sep 17 17:59:34.745073 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.744998    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38ff75e4bfd711cca31205f2faa1b2c15dc3c21fd93eb56c23ed16eb3ddf65f2"} err="failed to get container status \"38ff75e4bfd711cca31205f2faa1b2c15dc3c21fd93eb56c23ed16eb3ddf65f2\": rpc error: code = NotFound desc = could not find container \"38ff75e4bfd711cca31205f2faa1b2c15dc3c21fd93eb56c23ed16eb3ddf65f2\": container with ID starting with 38ff75e4bfd711cca31205f2faa1b2c15dc3c21fd93eb56c23ed16eb3ddf65f2 not found: ID does not exist"
Sep 17 17:59:34.830049 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.830022    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7m2j9\" (UniqueName: \"kubernetes.io/projected/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-kube-api-access-7m2j9\") pod \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\" (UID: \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\") "
Sep 17 17:59:34.830049 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.830050    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-console-config\") pod \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\" (UID: \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\") "
Sep 17 17:59:34.830268 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.830070    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-oauth-serving-cert\") pod \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\" (UID: \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\") "
Sep 17 17:59:34.830268 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.830100    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-service-ca\") pod \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\" (UID: \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\") "
Sep 17 17:59:34.830375 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.830277    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-trusted-ca-bundle\") pod \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\" (UID: \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\") "
Sep 17 17:59:34.830428 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.830390    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-console-oauth-config\") pod \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\" (UID: \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\") "
Sep 17 17:59:34.830428 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.830421    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-console-serving-cert\") pod \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\" (UID: \"01a3be75-f0e0-4691-8365-bf0b19b2aeb2\") "
Sep 17 17:59:34.830519 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.830459    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-service-ca" (OuterVolumeSpecName: "service-ca") pod "01a3be75-f0e0-4691-8365-bf0b19b2aeb2" (UID: "01a3be75-f0e0-4691-8365-bf0b19b2aeb2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:34.830594 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.830482    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-console-config" (OuterVolumeSpecName: "console-config") pod "01a3be75-f0e0-4691-8365-bf0b19b2aeb2" (UID: "01a3be75-f0e0-4691-8365-bf0b19b2aeb2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:34.830650 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.830594    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "01a3be75-f0e0-4691-8365-bf0b19b2aeb2" (UID: "01a3be75-f0e0-4691-8365-bf0b19b2aeb2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:34.830733 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.830716    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-console-config\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 17:59:34.830790 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.830741    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-oauth-serving-cert\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 17:59:34.830790 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.830755    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-service-ca\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 17:59:34.830946 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.830922    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "01a3be75-f0e0-4691-8365-bf0b19b2aeb2" (UID: "01a3be75-f0e0-4691-8365-bf0b19b2aeb2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:34.832687 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.832668    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "01a3be75-f0e0-4691-8365-bf0b19b2aeb2" (UID: "01a3be75-f0e0-4691-8365-bf0b19b2aeb2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:59:34.832906 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.832884    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "01a3be75-f0e0-4691-8365-bf0b19b2aeb2" (UID: "01a3be75-f0e0-4691-8365-bf0b19b2aeb2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:59:34.832961 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.832903    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-kube-api-access-7m2j9" (OuterVolumeSpecName: "kube-api-access-7m2j9") pod "01a3be75-f0e0-4691-8365-bf0b19b2aeb2" (UID: "01a3be75-f0e0-4691-8365-bf0b19b2aeb2"). InnerVolumeSpecName "kube-api-access-7m2j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:59:34.931261 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.931203    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-trusted-ca-bundle\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 17:59:34.931261 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.931224    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-console-oauth-config\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 17:59:34.931261 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.931234    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-console-serving-cert\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 17:59:34.931261 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:34.931242    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m2j9\" (UniqueName: \"kubernetes.io/projected/01a3be75-f0e0-4691-8365-bf0b19b2aeb2-kube-api-access-7m2j9\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 17:59:35.059464 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:35.059421    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-595f6ff97f-8xv29"]
Sep 17 17:59:35.061012 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:35.060990    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-595f6ff97f-8xv29"]
Sep 17 17:59:35.738811 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:35.738784    2581 generic.go:356] "Generic (PLEG): container finished" podID="184b0189-e1d0-4bfd-b970-596373fd0745" containerID="160752559f74b97eb0cb4dae6fc24dd395d56a89602288245ce49d905b5c9564" exitCode=0
Sep 17 17:59:35.739169 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:35.738859    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-cfm6h" event={"ID":"184b0189-e1d0-4bfd-b970-596373fd0745","Type":"ContainerDied","Data":"160752559f74b97eb0cb4dae6fc24dd395d56a89602288245ce49d905b5c9564"}
Sep 17 17:59:35.739232 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:35.739208    2581 scope.go:117] "RemoveContainer" containerID="160752559f74b97eb0cb4dae6fc24dd395d56a89602288245ce49d905b5c9564"
Sep 17 17:59:36.048325 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:36.048248    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01a3be75-f0e0-4691-8365-bf0b19b2aeb2" path="/var/lib/kubelet/pods/01a3be75-f0e0-4691-8365-bf0b19b2aeb2/volumes"
Sep 17 17:59:36.746026 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:36.745991    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-cfm6h" event={"ID":"184b0189-e1d0-4bfd-b970-596373fd0745","Type":"ContainerStarted","Data":"2f69619ddc498e45a7909375a0f046ffbb2869e53a5df7e5652142571900e3d0"}
Sep 17 17:59:56.731292 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:56.731211    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:59:56.750357 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:56.750337    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:59:56.834176 ip-10-0-134-209 kubenswrapper[2581]: I0917 17:59:56.834136    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:14.813418 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:14.813384    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:00:14.813895 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:14.813658    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerName="prometheus" containerID="cri-o://ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace" gracePeriod=600
Sep 17 18:00:14.813895 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:14.813685    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerName="kube-rbac-proxy" containerID="cri-o://5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6" gracePeriod=600
Sep 17 18:00:14.813895 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:14.813719    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerName="thanos-sidecar" containerID="cri-o://f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891" gracePeriod=600
Sep 17 18:00:14.813895 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:14.813761    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerName="kube-rbac-proxy-web" containerID="cri-o://8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935" gracePeriod=600
Sep 17 18:00:14.813895 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:14.813726    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerName="config-reloader" containerID="cri-o://5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837" gracePeriod=600
Sep 17 18:00:14.813895 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:14.813717    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerName="kube-rbac-proxy-thanos" containerID="cri-o://20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05" gracePeriod=600
Sep 17 18:00:15.061782 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.061752    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:15.150390 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.150361    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") "
Sep 17 18:00:15.150390 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.150391    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/744154b5-f9e0-45f5-a35f-f522ef3f15a4-config-out\") pod \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") "
Sep 17 18:00:15.150608 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.150413    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-thanos-prometheus-http-client-file\") pod \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") "
Sep 17 18:00:15.150608 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.150431    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-metrics-client-certs\") pod \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") "
Sep 17 18:00:15.150608 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.150453    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-configmap-metrics-client-ca\") pod \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") "
Sep 17 18:00:15.150608 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.150569    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-web-config\") pod \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") "
Sep 17 18:00:15.150807 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.150609    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-config\") pod \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") "
Sep 17 18:00:15.150807 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.150635    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/744154b5-f9e0-45f5-a35f-f522ef3f15a4-tls-assets\") pod \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") "
Sep 17 18:00:15.150807 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.150667    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/744154b5-f9e0-45f5-a35f-f522ef3f15a4-prometheus-k8s-db\") pod \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") "
Sep 17 18:00:15.150807 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.150699    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-configmap-serving-certs-ca-bundle\") pod \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") "
Sep 17 18:00:15.150807 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.150741    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") "
Sep 17 18:00:15.150807 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.150770    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-prometheus-k8s-rulefiles-1\") pod \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") "
Sep 17 18:00:15.151093 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.150824    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-prometheus-trusted-ca-bundle\") pod \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") "
Sep 17 18:00:15.151093 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.150831    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "744154b5-f9e0-45f5-a35f-f522ef3f15a4" (UID: "744154b5-f9e0-45f5-a35f-f522ef3f15a4"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:15.151093 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.150847    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-grpc-tls\") pod \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") "
Sep 17 18:00:15.151093 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.150889    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-kube-rbac-proxy\") pod \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") "
Sep 17 18:00:15.151093 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.150913    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqpsr\" (UniqueName: \"kubernetes.io/projected/744154b5-f9e0-45f5-a35f-f522ef3f15a4-kube-api-access-bqpsr\") pod \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") "
Sep 17 18:00:15.151093 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.150937    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-configmap-kubelet-serving-ca-bundle\") pod \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") "
Sep 17 18:00:15.151093 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.150974    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-prometheus-k8s-tls\") pod \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") "
Sep 17 18:00:15.151093 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.151009    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-prometheus-k8s-rulefiles-0\") pod \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") "
Sep 17 18:00:15.151093 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.151048    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-prometheus-k8s-rulefiles-2\") pod \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\" (UID: \"744154b5-f9e0-45f5-a35f-f522ef3f15a4\") "
Sep 17 18:00:15.151557 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.151280    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "744154b5-f9e0-45f5-a35f-f522ef3f15a4" (UID: "744154b5-f9e0-45f5-a35f-f522ef3f15a4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:15.151557 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.151367    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:15.151557 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.151387    2581 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-configmap-metrics-client-ca\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:15.151709 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.151628    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "744154b5-f9e0-45f5-a35f-f522ef3f15a4" (UID: "744154b5-f9e0-45f5-a35f-f522ef3f15a4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:15.151969 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.151944    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "744154b5-f9e0-45f5-a35f-f522ef3f15a4" (UID: "744154b5-f9e0-45f5-a35f-f522ef3f15a4"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:15.152108 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.152071    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "744154b5-f9e0-45f5-a35f-f522ef3f15a4" (UID: "744154b5-f9e0-45f5-a35f-f522ef3f15a4"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:15.153638 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.153608    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744154b5-f9e0-45f5-a35f-f522ef3f15a4-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "744154b5-f9e0-45f5-a35f-f522ef3f15a4" (UID: "744154b5-f9e0-45f5-a35f-f522ef3f15a4"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:15.153638 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.153610    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "744154b5-f9e0-45f5-a35f-f522ef3f15a4" (UID: "744154b5-f9e0-45f5-a35f-f522ef3f15a4"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:15.153991 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.153956    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "744154b5-f9e0-45f5-a35f-f522ef3f15a4" (UID: "744154b5-f9e0-45f5-a35f-f522ef3f15a4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:15.154100 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.153987    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "744154b5-f9e0-45f5-a35f-f522ef3f15a4" (UID: "744154b5-f9e0-45f5-a35f-f522ef3f15a4"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:15.154689 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.154646    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "744154b5-f9e0-45f5-a35f-f522ef3f15a4" (UID: "744154b5-f9e0-45f5-a35f-f522ef3f15a4"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:15.155906 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.155878    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744154b5-f9e0-45f5-a35f-f522ef3f15a4-config-out" (OuterVolumeSpecName: "config-out") pod "744154b5-f9e0-45f5-a35f-f522ef3f15a4" (UID: "744154b5-f9e0-45f5-a35f-f522ef3f15a4"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:15.156042 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.156022    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-config" (OuterVolumeSpecName: "config") pod "744154b5-f9e0-45f5-a35f-f522ef3f15a4" (UID: "744154b5-f9e0-45f5-a35f-f522ef3f15a4"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:15.156120 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.156071    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "744154b5-f9e0-45f5-a35f-f522ef3f15a4" (UID: "744154b5-f9e0-45f5-a35f-f522ef3f15a4"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:15.156574 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.156539    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "744154b5-f9e0-45f5-a35f-f522ef3f15a4" (UID: "744154b5-f9e0-45f5-a35f-f522ef3f15a4"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:15.156574 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.156552    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "744154b5-f9e0-45f5-a35f-f522ef3f15a4" (UID: "744154b5-f9e0-45f5-a35f-f522ef3f15a4"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:15.156574 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.156562    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/744154b5-f9e0-45f5-a35f-f522ef3f15a4-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "744154b5-f9e0-45f5-a35f-f522ef3f15a4" (UID: "744154b5-f9e0-45f5-a35f-f522ef3f15a4"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:00:15.156725 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.156582    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "744154b5-f9e0-45f5-a35f-f522ef3f15a4" (UID: "744154b5-f9e0-45f5-a35f-f522ef3f15a4"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:15.156792 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.156771    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/744154b5-f9e0-45f5-a35f-f522ef3f15a4-kube-api-access-bqpsr" (OuterVolumeSpecName: "kube-api-access-bqpsr") pod "744154b5-f9e0-45f5-a35f-f522ef3f15a4" (UID: "744154b5-f9e0-45f5-a35f-f522ef3f15a4"). InnerVolumeSpecName "kube-api-access-bqpsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:00:15.157587 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.157566    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "744154b5-f9e0-45f5-a35f-f522ef3f15a4" (UID: "744154b5-f9e0-45f5-a35f-f522ef3f15a4"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:15.167764 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.167738    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-web-config" (OuterVolumeSpecName: "web-config") pod "744154b5-f9e0-45f5-a35f-f522ef3f15a4" (UID: "744154b5-f9e0-45f5-a35f-f522ef3f15a4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:15.252256 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.252237    2581 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-config\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:15.252256 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.252256    2581 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/744154b5-f9e0-45f5-a35f-f522ef3f15a4-tls-assets\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:15.252371 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.252265    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/744154b5-f9e0-45f5-a35f-f522ef3f15a4-prometheus-k8s-db\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:15.252371 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.252274    2581 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:15.252371 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.252282    2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:15.252371 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.252291    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:15.252371 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.252298    2581 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-grpc-tls\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:15.252371 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.252308    2581 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-kube-rbac-proxy\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:15.252371 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.252315    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqpsr\" (UniqueName: \"kubernetes.io/projected/744154b5-f9e0-45f5-a35f-f522ef3f15a4-kube-api-access-bqpsr\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:15.252371 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.252325    2581 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:15.252371 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.252333    2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:15.252371 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.252341    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:15.252371 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.252348    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/744154b5-f9e0-45f5-a35f-f522ef3f15a4-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:15.252371 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.252356    2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:15.252371 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.252366    2581 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/744154b5-f9e0-45f5-a35f-f522ef3f15a4-config-out\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:15.252371 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.252374    2581 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:15.252793 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.252383    2581 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-secret-metrics-client-certs\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:15.252793 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.252410    2581 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/744154b5-f9e0-45f5-a35f-f522ef3f15a4-web-config\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:15.897984 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.897957    2581 generic.go:356] "Generic (PLEG): container finished" podID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerID="20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05" exitCode=0
Sep 17 18:00:15.897984 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.897982    2581 generic.go:356] "Generic (PLEG): container finished" podID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerID="5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6" exitCode=0
Sep 17 18:00:15.897984 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.897988    2581 generic.go:356] "Generic (PLEG): container finished" podID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerID="8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935" exitCode=0
Sep 17 18:00:15.898471 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.897996    2581 generic.go:356] "Generic (PLEG): container finished" podID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerID="f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891" exitCode=0
Sep 17 18:00:15.898471 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.898001    2581 generic.go:356] "Generic (PLEG): container finished" podID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerID="5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837" exitCode=0
Sep 17 18:00:15.898471 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.898006    2581 generic.go:356] "Generic (PLEG): container finished" podID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerID="ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace" exitCode=0
Sep 17 18:00:15.898471 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.898031    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:15.898471 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.898043    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"744154b5-f9e0-45f5-a35f-f522ef3f15a4","Type":"ContainerDied","Data":"20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05"}
Sep 17 18:00:15.898471 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.898135    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"744154b5-f9e0-45f5-a35f-f522ef3f15a4","Type":"ContainerDied","Data":"5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6"}
Sep 17 18:00:15.898471 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.898172    2581 scope.go:117] "RemoveContainer" containerID="20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05"
Sep 17 18:00:15.898471 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.898197    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"744154b5-f9e0-45f5-a35f-f522ef3f15a4","Type":"ContainerDied","Data":"8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935"}
Sep 17 18:00:15.898471 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.898240    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"744154b5-f9e0-45f5-a35f-f522ef3f15a4","Type":"ContainerDied","Data":"f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891"}
Sep 17 18:00:15.898471 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.898280    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"744154b5-f9e0-45f5-a35f-f522ef3f15a4","Type":"ContainerDied","Data":"5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837"}
Sep 17 18:00:15.898471 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.898317    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"744154b5-f9e0-45f5-a35f-f522ef3f15a4","Type":"ContainerDied","Data":"ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace"}
Sep 17 18:00:15.898471 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.898355    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"744154b5-f9e0-45f5-a35f-f522ef3f15a4","Type":"ContainerDied","Data":"fcab4654017dc3b72bfc4e1064df57727ad814aeec4a6a5d627177fb84459262"}
Sep 17 18:00:15.906085 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.906065    2581 scope.go:117] "RemoveContainer" containerID="5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6"
Sep 17 18:00:15.912748 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.912733    2581 scope.go:117] "RemoveContainer" containerID="8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935"
Sep 17 18:00:15.918940 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.918925    2581 scope.go:117] "RemoveContainer" containerID="f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891"
Sep 17 18:00:15.924341 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.924321    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:00:15.925693 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.925679    2581 scope.go:117] "RemoveContainer" containerID="5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837"
Sep 17 18:00:15.930873 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.930853    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:00:15.932814 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.932799    2581 scope.go:117] "RemoveContainer" containerID="ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace"
Sep 17 18:00:15.939641 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.939627    2581 scope.go:117] "RemoveContainer" containerID="962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f"
Sep 17 18:00:15.945789 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.945775    2581 scope.go:117] "RemoveContainer" containerID="20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05"
Sep 17 18:00:15.946043 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:00:15.946026    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05\": container with ID starting with 20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05 not found: ID does not exist" containerID="20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05"
Sep 17 18:00:15.946088 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.946051    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05"} err="failed to get container status \"20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05\": rpc error: code = NotFound desc = could not find container \"20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05\": container with ID starting with 20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05 not found: ID does not exist"
Sep 17 18:00:15.946088 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.946069    2581 scope.go:117] "RemoveContainer" containerID="5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6"
Sep 17 18:00:15.946318 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:00:15.946304    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6\": container with ID starting with 5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6 not found: ID does not exist" containerID="5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6"
Sep 17 18:00:15.946353 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.946321    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6"} err="failed to get container status \"5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6\": rpc error: code = NotFound desc = could not find container \"5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6\": container with ID starting with 5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6 not found: ID does not exist"
Sep 17 18:00:15.946353 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.946332    2581 scope.go:117] "RemoveContainer" containerID="8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935"
Sep 17 18:00:15.946541 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:00:15.946526    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935\": container with ID starting with 8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935 not found: ID does not exist" containerID="8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935"
Sep 17 18:00:15.946581 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.946546    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935"} err="failed to get container status \"8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935\": rpc error: code = NotFound desc = could not find container \"8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935\": container with ID starting with 8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935 not found: ID does not exist"
Sep 17 18:00:15.946581 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.946558    2581 scope.go:117] "RemoveContainer" containerID="f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891"
Sep 17 18:00:15.946749 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:00:15.946735    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891\": container with ID starting with f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891 not found: ID does not exist" containerID="f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891"
Sep 17 18:00:15.946784 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.946751    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891"} err="failed to get container status \"f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891\": rpc error: code = NotFound desc = could not find container \"f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891\": container with ID starting with f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891 not found: ID does not exist"
Sep 17 18:00:15.946784 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.946762    2581 scope.go:117] "RemoveContainer" containerID="5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837"
Sep 17 18:00:15.946950 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:00:15.946936    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837\": container with ID starting with 5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837 not found: ID does not exist" containerID="5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837"
Sep 17 18:00:15.946992 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.946953    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837"} err="failed to get container status \"5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837\": rpc error: code = NotFound desc = could not find container \"5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837\": container with ID starting with 5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837 not found: ID does not exist"
Sep 17 18:00:15.946992 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.946965    2581 scope.go:117] "RemoveContainer" containerID="ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace"
Sep 17 18:00:15.947189 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:00:15.947139    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace\": container with ID starting with ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace not found: ID does not exist" containerID="ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace"
Sep 17 18:00:15.947232 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.947201    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace"} err="failed to get container status \"ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace\": rpc error: code = NotFound desc = could not find container \"ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace\": container with ID starting with ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace not found: ID does not exist"
Sep 17 18:00:15.947232 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.947222    2581 scope.go:117] "RemoveContainer" containerID="962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f"
Sep 17 18:00:15.947462 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:00:15.947445    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f\": container with ID starting with 962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f not found: ID does not exist" containerID="962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f"
Sep 17 18:00:15.947498 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.947475    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f"} err="failed to get container status \"962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f\": rpc error: code = NotFound desc = could not find container \"962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f\": container with ID starting with 962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f not found: ID does not exist"
Sep 17 18:00:15.947498 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.947490    2581 scope.go:117] "RemoveContainer" containerID="20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05"
Sep 17 18:00:15.947724 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.947708    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05"} err="failed to get container status \"20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05\": rpc error: code = NotFound desc = could not find container \"20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05\": container with ID starting with 20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05 not found: ID does not exist"
Sep 17 18:00:15.947766 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.947724    2581 scope.go:117] "RemoveContainer" containerID="5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6"
Sep 17 18:00:15.947945 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.947927    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6"} err="failed to get container status \"5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6\": rpc error: code = NotFound desc = could not find container \"5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6\": container with ID starting with 5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6 not found: ID does not exist"
Sep 17 18:00:15.948047 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.947948    2581 scope.go:117] "RemoveContainer" containerID="8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935"
Sep 17 18:00:15.948230 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.948212    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935"} err="failed to get container status \"8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935\": rpc error: code = NotFound desc = could not find container \"8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935\": container with ID starting with 8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935 not found: ID does not exist"
Sep 17 18:00:15.948280 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.948231    2581 scope.go:117] "RemoveContainer" containerID="f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891"
Sep 17 18:00:15.948431 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.948415    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891"} err="failed to get container status \"f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891\": rpc error: code = NotFound desc = could not find container \"f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891\": container with ID starting with f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891 not found: ID does not exist"
Sep 17 18:00:15.948472 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.948431    2581 scope.go:117] "RemoveContainer" containerID="5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837"
Sep 17 18:00:15.948651 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.948637    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837"} err="failed to get container status \"5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837\": rpc error: code = NotFound desc = could not find container \"5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837\": container with ID starting with 5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837 not found: ID does not exist"
Sep 17 18:00:15.948695 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.948652    2581 scope.go:117] "RemoveContainer" containerID="ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace"
Sep 17 18:00:15.948839 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.948825    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace"} err="failed to get container status \"ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace\": rpc error: code = NotFound desc = could not find container \"ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace\": container with ID starting with ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace not found: ID does not exist"
Sep 17 18:00:15.948883 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.948839    2581 scope.go:117] "RemoveContainer" containerID="962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f"
Sep 17 18:00:15.949030 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.949014    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f"} err="failed to get container status \"962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f\": rpc error: code = NotFound desc = could not find container \"962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f\": container with ID starting with 962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f not found: ID does not exist"
Sep 17 18:00:15.949071 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.949033    2581 scope.go:117] "RemoveContainer" containerID="20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05"
Sep 17 18:00:15.949256 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.949230    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05"} err="failed to get container status \"20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05\": rpc error: code = NotFound desc = could not find container \"20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05\": container with ID starting with 20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05 not found: ID does not exist"
Sep 17 18:00:15.949330 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.949256    2581 scope.go:117] "RemoveContainer" containerID="5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6"
Sep 17 18:00:15.949514 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.949497    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6"} err="failed to get container status \"5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6\": rpc error: code = NotFound desc = could not find container \"5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6\": container with ID starting with 5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6 not found: ID does not exist"
Sep 17 18:00:15.949561 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.949516    2581 scope.go:117] "RemoveContainer" containerID="8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935"
Sep 17 18:00:15.949733 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.949715    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935"} err="failed to get container status \"8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935\": rpc error: code = NotFound desc = could not find container \"8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935\": container with ID starting with 8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935 not found: ID does not exist"
Sep 17 18:00:15.949777 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.949734    2581 scope.go:117] "RemoveContainer" containerID="f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891"
Sep 17 18:00:15.949925 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.949907    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891"} err="failed to get container status \"f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891\": rpc error: code = NotFound desc = could not find container \"f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891\": container with ID starting with f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891 not found: ID does not exist"
Sep 17 18:00:15.949988 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.949926    2581 scope.go:117] "RemoveContainer" containerID="5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837"
Sep 17 18:00:15.950174 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.950140    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837"} err="failed to get container status \"5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837\": rpc error: code = NotFound desc = could not find container \"5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837\": container with ID starting with 5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837 not found: ID does not exist"
Sep 17 18:00:15.950216 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.950175    2581 scope.go:117] "RemoveContainer" containerID="ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace"
Sep 17 18:00:15.950393 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.950376    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace"} err="failed to get container status \"ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace\": rpc error: code = NotFound desc = could not find container \"ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace\": container with ID starting with ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace not found: ID does not exist"
Sep 17 18:00:15.950438 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.950393    2581 scope.go:117] "RemoveContainer" containerID="962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f"
Sep 17 18:00:15.950587 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.950571    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f"} err="failed to get container status \"962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f\": rpc error: code = NotFound desc = could not find container \"962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f\": container with ID starting with 962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f not found: ID does not exist"
Sep 17 18:00:15.950650 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.950589    2581 scope.go:117] "RemoveContainer" containerID="20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05"
Sep 17 18:00:15.950818 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.950802    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05"} err="failed to get container status \"20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05\": rpc error: code = NotFound desc = could not find container \"20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05\": container with ID starting with 20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05 not found: ID does not exist"
Sep 17 18:00:15.950861 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.950819    2581 scope.go:117] "RemoveContainer" containerID="5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6"
Sep 17 18:00:15.951022 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.951005    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6"} err="failed to get container status \"5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6\": rpc error: code = NotFound desc = could not find container \"5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6\": container with ID starting with 5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6 not found: ID does not exist"
Sep 17 18:00:15.951088 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.951024    2581 scope.go:117] "RemoveContainer" containerID="8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935"
Sep 17 18:00:15.951252 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.951229    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935"} err="failed to get container status \"8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935\": rpc error: code = NotFound desc = could not find container \"8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935\": container with ID starting with 8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935 not found: ID does not exist"
Sep 17 18:00:15.951252 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.951252    2581 scope.go:117] "RemoveContainer" containerID="f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891"
Sep 17 18:00:15.951465 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.951447    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891"} err="failed to get container status \"f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891\": rpc error: code = NotFound desc = could not find container \"f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891\": container with ID starting with f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891 not found: ID does not exist"
Sep 17 18:00:15.951517 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.951466    2581 scope.go:117] "RemoveContainer" containerID="5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837"
Sep 17 18:00:15.951671 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.951654    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837"} err="failed to get container status \"5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837\": rpc error: code = NotFound desc = could not find container \"5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837\": container with ID starting with 5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837 not found: ID does not exist"
Sep 17 18:00:15.951713 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.951672    2581 scope.go:117] "RemoveContainer" containerID="ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace"
Sep 17 18:00:15.951920 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.951903    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace"} err="failed to get container status \"ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace\": rpc error: code = NotFound desc = could not find container \"ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace\": container with ID starting with ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace not found: ID does not exist"
Sep 17 18:00:15.951991 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.951922    2581 scope.go:117] "RemoveContainer" containerID="962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f"
Sep 17 18:00:15.952193 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.952176    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f"} err="failed to get container status \"962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f\": rpc error: code = NotFound desc = could not find container \"962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f\": container with ID starting with 962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f not found: ID does not exist"
Sep 17 18:00:15.952243 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.952194    2581 scope.go:117] "RemoveContainer" containerID="20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05"
Sep 17 18:00:15.952393 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.952376    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05"} err="failed to get container status \"20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05\": rpc error: code = NotFound desc = could not find container \"20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05\": container with ID starting with 20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05 not found: ID does not exist"
Sep 17 18:00:15.952456 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.952395    2581 scope.go:117] "RemoveContainer" containerID="5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6"
Sep 17 18:00:15.952596 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.952581    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6"} err="failed to get container status \"5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6\": rpc error: code = NotFound desc = could not find container \"5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6\": container with ID starting with 5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6 not found: ID does not exist"
Sep 17 18:00:15.952640 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.952596    2581 scope.go:117] "RemoveContainer" containerID="8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935"
Sep 17 18:00:15.952800 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.952783    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935"} err="failed to get container status \"8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935\": rpc error: code = NotFound desc = could not find container \"8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935\": container with ID starting with 8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935 not found: ID does not exist"
Sep 17 18:00:15.952869 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.952802    2581 scope.go:117] "RemoveContainer" containerID="f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891"
Sep 17 18:00:15.953033 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.953017    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891"} err="failed to get container status \"f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891\": rpc error: code = NotFound desc = could not find container \"f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891\": container with ID starting with f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891 not found: ID does not exist"
Sep 17 18:00:15.953080 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.953034    2581 scope.go:117] "RemoveContainer" containerID="5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837"
Sep 17 18:00:15.953263 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.953239    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837"} err="failed to get container status \"5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837\": rpc error: code = NotFound desc = could not find container \"5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837\": container with ID starting with 5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837 not found: ID does not exist"
Sep 17 18:00:15.953332 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.953264    2581 scope.go:117] "RemoveContainer" containerID="ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace"
Sep 17 18:00:15.953482 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.953465    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace"} err="failed to get container status \"ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace\": rpc error: code = NotFound desc = could not find container \"ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace\": container with ID starting with ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace not found: ID does not exist"
Sep 17 18:00:15.953529 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.953483    2581 scope.go:117] "RemoveContainer" containerID="962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f"
Sep 17 18:00:15.953691 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.953675    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f"} err="failed to get container status \"962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f\": rpc error: code = NotFound desc = could not find container \"962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f\": container with ID starting with 962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f not found: ID does not exist"
Sep 17 18:00:15.953756 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.953692    2581 scope.go:117] "RemoveContainer" containerID="20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05"
Sep 17 18:00:15.953928 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.953912    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05"} err="failed to get container status \"20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05\": rpc error: code = NotFound desc = could not find container \"20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05\": container with ID starting with 20c55d6f632a66e59079423f2400a7bad5ac73118fc277080e11ea5b0dac7c05 not found: ID does not exist"
Sep 17 18:00:15.953970 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.953929    2581 scope.go:117] "RemoveContainer" containerID="5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6"
Sep 17 18:00:15.954129 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.954113    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6"} err="failed to get container status \"5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6\": rpc error: code = NotFound desc = could not find container \"5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6\": container with ID starting with 5b8c8aaf52a78818874888ae05d6545dceef7eec66d6c7ba27df3c8cd9651fa6 not found: ID does not exist"
Sep 17 18:00:15.954220 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.954130    2581 scope.go:117] "RemoveContainer" containerID="8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935"
Sep 17 18:00:15.954353 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.954336    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935"} err="failed to get container status \"8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935\": rpc error: code = NotFound desc = could not find container \"8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935\": container with ID starting with 8ecc838527972b63ed21801b3861645f1e58f0ff385363ec38a5baf14d4e9935 not found: ID does not exist"
Sep 17 18:00:15.954394 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.954354    2581 scope.go:117] "RemoveContainer" containerID="f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891"
Sep 17 18:00:15.954551 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.954535    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891"} err="failed to get container status \"f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891\": rpc error: code = NotFound desc = could not find container \"f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891\": container with ID starting with f4459d615458a02b0936260571e732730c5025d7e48621c57164c84836630891 not found: ID does not exist"
Sep 17 18:00:15.954608 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.954552    2581 scope.go:117] "RemoveContainer" containerID="5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837"
Sep 17 18:00:15.954748 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.954726    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837"} err="failed to get container status \"5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837\": rpc error: code = NotFound desc = could not find container \"5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837\": container with ID starting with 5209e29f9cf87e8f8a5845c2c3695450751134b4d1bc2b4f5612b6781c90e837 not found: ID does not exist"
Sep 17 18:00:15.954791 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.954749    2581 scope.go:117] "RemoveContainer" containerID="ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace"
Sep 17 18:00:15.954923 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.954905    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace"} err="failed to get container status \"ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace\": rpc error: code = NotFound desc = could not find container \"ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace\": container with ID starting with ee4caad63507a74aa7fafe268dccceeb061195535567bb9875c2b695c7855ace not found: ID does not exist"
Sep 17 18:00:15.954986 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.954924    2581 scope.go:117] "RemoveContainer" containerID="962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f"
Sep 17 18:00:15.955106 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.955091    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f"} err="failed to get container status \"962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f\": rpc error: code = NotFound desc = could not find container \"962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f\": container with ID starting with 962e71212617ef2690b79ecd5a04d84007dbc4ace3690140c18adc91cef9d34f not found: ID does not exist"
Sep 17 18:00:15.970772 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.970752    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:00:15.971017 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.971005    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerName="prometheus"
Sep 17 18:00:15.971057 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.971019    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerName="prometheus"
Sep 17 18:00:15.971057 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.971027    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerName="kube-rbac-proxy"
Sep 17 18:00:15.971057 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.971032    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerName="kube-rbac-proxy"
Sep 17 18:00:15.971057 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.971047    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01a3be75-f0e0-4691-8365-bf0b19b2aeb2" containerName="console"
Sep 17 18:00:15.971057 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.971053    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="01a3be75-f0e0-4691-8365-bf0b19b2aeb2" containerName="console"
Sep 17 18:00:15.971218 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.971060    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerName="init-config-reloader"
Sep 17 18:00:15.971218 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.971065    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerName="init-config-reloader"
Sep 17 18:00:15.971218 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.971074    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerName="config-reloader"
Sep 17 18:00:15.971218 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.971078    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerName="config-reloader"
Sep 17 18:00:15.971218 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.971085    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerName="kube-rbac-proxy-web"
Sep 17 18:00:15.971218 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.971090    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerName="kube-rbac-proxy-web"
Sep 17 18:00:15.971218 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.971095    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerName="kube-rbac-proxy-thanos"
Sep 17 18:00:15.971218 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.971100    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerName="kube-rbac-proxy-thanos"
Sep 17 18:00:15.971218 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.971108    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerName="thanos-sidecar"
Sep 17 18:00:15.971218 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.971113    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerName="thanos-sidecar"
Sep 17 18:00:15.971218 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.971178    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerName="config-reloader"
Sep 17 18:00:15.971218 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.971186    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerName="kube-rbac-proxy"
Sep 17 18:00:15.971218 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.971192    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerName="kube-rbac-proxy-thanos"
Sep 17 18:00:15.971218 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.971196    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="01a3be75-f0e0-4691-8365-bf0b19b2aeb2" containerName="console"
Sep 17 18:00:15.971218 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.971202    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerName="thanos-sidecar"
Sep 17 18:00:15.971218 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.971206    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerName="prometheus"
Sep 17 18:00:15.971218 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.971211    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" containerName="kube-rbac-proxy-web"
Sep 17 18:00:15.976122 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.976103    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:15.979909 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.979892    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 18:00:15.980449 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.980434    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 18:00:15.980501 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.980434    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 18:00:15.980547 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.980499    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 18:00:15.980752 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.980736    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 18:00:15.980893 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.980841    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 18:00:15.980893 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.980859    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 18:00:15.981186 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.981168    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 18:00:15.981533 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.981514    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 18:00:15.981618 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.981519    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-s8jdl\""
Sep 17 18:00:15.981618 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.981610    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 18:00:15.981741 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.981727    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 18:00:15.982697 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.982545    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 18:00:15.982858 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.982817    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-26otalvshcc79\""
Sep 17 18:00:15.983569 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.983551    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 18:00:15.990883 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:15.990861    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 18:00:16.001949 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.001929    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:00:16.048434 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.048409    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="744154b5-f9e0-45f5-a35f-f522ef3f15a4" path="/var/lib/kubelet/pods/744154b5-f9e0-45f5-a35f-f522ef3f15a4/volumes"
Sep 17 18:00:16.058741 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.058718    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.058822 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.058744    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.058822 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.058763    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-config\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.058822 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.058777    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.058822 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.058805    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.058944 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.058830    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.058944 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.058849    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfhj7\" (UniqueName: \"kubernetes.io/projected/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-kube-api-access-cfhj7\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.058944 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.058865    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.058944 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.058917    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.058944 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.058932    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.059079 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.058945    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.059079 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.058988    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.059079 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.059003    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.059079 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.059017    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.059079 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.059036    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.059079 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.059071    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-config-out\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.059303 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.059099    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.059303 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.059122    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.059303 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.059138    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.059303 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.059176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-web-config\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.160189 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.160168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.160289 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.160194    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.160289 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.160210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.160289 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.160228    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.160289 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.160243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.160465 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.160388    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.160465 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.160426    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.160465 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.160448    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-config-out\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.160603 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.160466    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.160603 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.160486    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.160603 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.160507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.160603 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.160530    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-web-config\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.160603 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.160557    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.160603 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.160584    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.160877 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.160613    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-config\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.160877 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.160636    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.160877 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.160659    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.160877 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.160687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.160877 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.160724    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfhj7\" (UniqueName: \"kubernetes.io/projected/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-kube-api-access-cfhj7\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.160877 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.160750    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.160877 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.160843    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.161405 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.161132    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.161457 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.161433    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.161515 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.161485    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.161664 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.161642    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.163599 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.163305    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-config-out\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.163695 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.163653    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.164380 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.163893    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.164380 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.164278    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.164380 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.164326    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.164567 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.164394    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.164567 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.164414    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.164849 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.164815    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-config\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.164946 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.164817    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-web-config\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.165198 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.165179    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.166113 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.166092    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.166452 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.166433    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.166806 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.166784    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.167373 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.167357    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.173784 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.173764    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfhj7\" (UniqueName: \"kubernetes.io/projected/43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d-kube-api-access-cfhj7\") pod \"prometheus-k8s-0\" (UID: \"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.285791 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.285763    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:00:16.569051 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.568974    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:00:16.571952 ip-10-0-134-209 kubenswrapper[2581]: W0917 18:00:16.571930    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a7f567_0cf3_4f0f_90f4_b5cdbb057a4d.slice/crio-f488f94214f9d4d65b53110373ea2c2a525478058d78ca5de38b9f0432f3850d WatchSource:0}: Error finding container f488f94214f9d4d65b53110373ea2c2a525478058d78ca5de38b9f0432f3850d: Status 404 returned error can't find the container with id f488f94214f9d4d65b53110373ea2c2a525478058d78ca5de38b9f0432f3850d
Sep 17 18:00:16.904443 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.904406    2581 generic.go:356] "Generic (PLEG): container finished" podID="43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d" containerID="69014550f927e238574cb75ca4b1dbb244a264f13b46709b62ddd05a3d0f6b3d" exitCode=0
Sep 17 18:00:16.904775 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.904475    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d","Type":"ContainerDied","Data":"69014550f927e238574cb75ca4b1dbb244a264f13b46709b62ddd05a3d0f6b3d"}
Sep 17 18:00:16.904775 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:16.904538    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d","Type":"ContainerStarted","Data":"f488f94214f9d4d65b53110373ea2c2a525478058d78ca5de38b9f0432f3850d"}
Sep 17 18:00:17.912690 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:17.912658    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d","Type":"ContainerStarted","Data":"f2528ca53ee31ddbbc32f5901a7667339a85ee03742ba7c41598742e115a70ba"}
Sep 17 18:00:17.913026 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:17.912718    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d","Type":"ContainerStarted","Data":"d646317f262ac887ed10646c65b413b809ebfdccd152d0abe8ff041240067cbd"}
Sep 17 18:00:17.913026 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:17.912744    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d","Type":"ContainerStarted","Data":"836b9967ac861b2f9941ee4cdfdd3430679ea34d77d36cc9c50ecc94e2681fa2"}
Sep 17 18:00:17.913026 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:17.912766    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d","Type":"ContainerStarted","Data":"fee3671303c5711a3d8ece35791ff21d0e3c3978002f2171e6eec3cefffc1eca"}
Sep 17 18:00:17.913026 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:17.912789    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d","Type":"ContainerStarted","Data":"2b6953b3114df462e2a389e2283501dd16e5fa8dc1ae5ed445b281307a468732"}
Sep 17 18:00:17.913026 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:17.912810    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d","Type":"ContainerStarted","Data":"adc4b4dde0e62bb9780bc63d9006be3aaa3365ed5b32da444e248d95bd3b1831"}
Sep 17 18:00:17.946121 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:17.946068    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.92695523 podStartE2EDuration="2.946047332s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.092102ms" isStatefulPod=false podCreationTimestamp="2026-09-17 18:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:17.944682862 +0000 UTC m=+200.045146681" watchObservedRunningTime="2026-09-17 18:00:17.946047332 +0000 UTC m=+200.046511152"
Sep 17 18:00:21.286634 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:00:21.286601    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:01:16.286181 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:01:16.286077    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:01:16.301316 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:01:16.301291    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:01:17.151129 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:01:17.151097    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:01:59.902416 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:01:59.902387    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm8hx_913ce1e9-9fb6-4049-a06d-9ebc69d3e528/ovn-acl-logging/0.log"
Sep 17 18:01:59.904069 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:01:59.904046    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm8hx_913ce1e9-9fb6-4049-a06d-9ebc69d3e528/ovn-acl-logging/0.log"
Sep 17 18:01:59.909664 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:01:59.909644    2581 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 18:04:55.893553 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:04:55.893516    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-xpktj"]
Sep 17 18:04:55.896924 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:04:55.896908    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-xpktj"
Sep 17 18:04:55.899229 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:04:55.899207    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 18:04:55.899357 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:04:55.899252    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-t6pqr\""
Sep 17 18:04:55.899851 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:04:55.899830    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 18:04:55.899851 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:04:55.899846    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 18:04:55.900175 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:04:55.900140    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 18:04:55.905931 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:04:55.905908    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-xpktj"]
Sep 17 18:04:56.014144 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:04:56.014116    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kclj8\" (UniqueName: \"kubernetes.io/projected/3f612f22-acc0-41bc-99e0-6db0a1eb887e-kube-api-access-kclj8\") pod \"s3-init-xpktj\" (UID: \"3f612f22-acc0-41bc-99e0-6db0a1eb887e\") " pod="kserve/s3-init-xpktj"
Sep 17 18:04:56.014296 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:04:56.014175    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f612f22-acc0-41bc-99e0-6db0a1eb887e-model-cache\") pod \"s3-init-xpktj\" (UID: \"3f612f22-acc0-41bc-99e0-6db0a1eb887e\") " pod="kserve/s3-init-xpktj"
Sep 17 18:04:56.115308 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:04:56.115270    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kclj8\" (UniqueName: \"kubernetes.io/projected/3f612f22-acc0-41bc-99e0-6db0a1eb887e-kube-api-access-kclj8\") pod \"s3-init-xpktj\" (UID: \"3f612f22-acc0-41bc-99e0-6db0a1eb887e\") " pod="kserve/s3-init-xpktj"
Sep 17 18:04:56.115308 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:04:56.115317    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f612f22-acc0-41bc-99e0-6db0a1eb887e-model-cache\") pod \"s3-init-xpktj\" (UID: \"3f612f22-acc0-41bc-99e0-6db0a1eb887e\") " pod="kserve/s3-init-xpktj"
Sep 17 18:04:56.115633 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:04:56.115618    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f612f22-acc0-41bc-99e0-6db0a1eb887e-model-cache\") pod \"s3-init-xpktj\" (UID: \"3f612f22-acc0-41bc-99e0-6db0a1eb887e\") " pod="kserve/s3-init-xpktj"
Sep 17 18:04:56.123999 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:04:56.123971    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kclj8\" (UniqueName: \"kubernetes.io/projected/3f612f22-acc0-41bc-99e0-6db0a1eb887e-kube-api-access-kclj8\") pod \"s3-init-xpktj\" (UID: \"3f612f22-acc0-41bc-99e0-6db0a1eb887e\") " pod="kserve/s3-init-xpktj"
Sep 17 18:04:56.218016 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:04:56.217947    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-xpktj"
Sep 17 18:04:56.489060 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:04:56.488984    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-xpktj"]
Sep 17 18:04:56.491919 ip-10-0-134-209 kubenswrapper[2581]: W0917 18:04:56.491883    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f612f22_acc0_41bc_99e0_6db0a1eb887e.slice/crio-e8bd044e7cb02d2dd4669ea117ee51f1d9389774bd41de18991850e69e797e93 WatchSource:0}: Error finding container e8bd044e7cb02d2dd4669ea117ee51f1d9389774bd41de18991850e69e797e93: Status 404 returned error can't find the container with id e8bd044e7cb02d2dd4669ea117ee51f1d9389774bd41de18991850e69e797e93
Sep 17 18:04:56.493750 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:04:56.493734    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:04:57.015878 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:04:57.015833    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-xpktj" event={"ID":"3f612f22-acc0-41bc-99e0-6db0a1eb887e","Type":"ContainerStarted","Data":"e8bd044e7cb02d2dd4669ea117ee51f1d9389774bd41de18991850e69e797e93"}
Sep 17 18:04:59.890303 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:04:59.890274    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 18:05:00.031298 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:05:00.031254    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-xpktj" event={"ID":"3f612f22-acc0-41bc-99e0-6db0a1eb887e","Type":"ContainerStarted","Data":"6d63d62c2f9789c7d9a26ddd2d8669b23f892c2b428cf4da900b46f88b54570d"}
Sep 17 18:05:02.040857 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:05:02.040826    2581 generic.go:356] "Generic (PLEG): container finished" podID="3f612f22-acc0-41bc-99e0-6db0a1eb887e" containerID="6d63d62c2f9789c7d9a26ddd2d8669b23f892c2b428cf4da900b46f88b54570d" exitCode=0
Sep 17 18:05:02.041341 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:05:02.040904    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-xpktj" event={"ID":"3f612f22-acc0-41bc-99e0-6db0a1eb887e","Type":"ContainerDied","Data":"6d63d62c2f9789c7d9a26ddd2d8669b23f892c2b428cf4da900b46f88b54570d"}
Sep 17 18:05:06.468990 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:05:06.468966    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 18:05:07.069666 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:05:07.069630    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-xpktj" event={"ID":"3f612f22-acc0-41bc-99e0-6db0a1eb887e","Type":"ContainerStarted","Data":"754a48bd89a5f1fa408cdeea3970efdbadff2dfc1c63846f0d86c02217ade335"}
Sep 17 18:05:07.101717 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:05:07.101668    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-xpktj" podStartSLOduration=2.642925271 podStartE2EDuration="12.101652508s" totalImagesPullingTime="7.81273218s" totalInitContainerRuntime="1.645995057s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:05:07.101290463 +0000 UTC m=+489.201754280" watchObservedRunningTime="2026-09-17 18:05:07.101652508 +0000 UTC m=+489.202116327"
Sep 17 18:06:11.327853 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:11.327822    2581 generic.go:356] "Generic (PLEG): container finished" podID="3f612f22-acc0-41bc-99e0-6db0a1eb887e" containerID="754a48bd89a5f1fa408cdeea3970efdbadff2dfc1c63846f0d86c02217ade335" exitCode=0
Sep 17 18:06:11.328225 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:11.327898    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-xpktj" event={"ID":"3f612f22-acc0-41bc-99e0-6db0a1eb887e","Type":"ContainerDied","Data":"754a48bd89a5f1fa408cdeea3970efdbadff2dfc1c63846f0d86c02217ade335"}
Sep 17 18:06:12.458777 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:12.458753    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-xpktj"
Sep 17 18:06:12.608913 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:12.608831    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f612f22-acc0-41bc-99e0-6db0a1eb887e-model-cache\") pod \"3f612f22-acc0-41bc-99e0-6db0a1eb887e\" (UID: \"3f612f22-acc0-41bc-99e0-6db0a1eb887e\") "
Sep 17 18:06:12.609060 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:12.608941    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kclj8\" (UniqueName: \"kubernetes.io/projected/3f612f22-acc0-41bc-99e0-6db0a1eb887e-kube-api-access-kclj8\") pod \"3f612f22-acc0-41bc-99e0-6db0a1eb887e\" (UID: \"3f612f22-acc0-41bc-99e0-6db0a1eb887e\") "
Sep 17 18:06:12.611327 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:12.611302    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f612f22-acc0-41bc-99e0-6db0a1eb887e-kube-api-access-kclj8" (OuterVolumeSpecName: "kube-api-access-kclj8") pod "3f612f22-acc0-41bc-99e0-6db0a1eb887e" (UID: "3f612f22-acc0-41bc-99e0-6db0a1eb887e"). InnerVolumeSpecName "kube-api-access-kclj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:06:12.630864 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:12.630840    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f612f22-acc0-41bc-99e0-6db0a1eb887e-model-cache" (OuterVolumeSpecName: "model-cache") pod "3f612f22-acc0-41bc-99e0-6db0a1eb887e" (UID: "3f612f22-acc0-41bc-99e0-6db0a1eb887e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:06:12.709589 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:12.709568    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kclj8\" (UniqueName: \"kubernetes.io/projected/3f612f22-acc0-41bc-99e0-6db0a1eb887e-kube-api-access-kclj8\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:06:12.709589 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:12.709588    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f612f22-acc0-41bc-99e0-6db0a1eb887e-model-cache\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:06:13.337850 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:13.337823    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-xpktj"
Sep 17 18:06:13.337850 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:13.337845    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-xpktj" event={"ID":"3f612f22-acc0-41bc-99e0-6db0a1eb887e","Type":"ContainerDied","Data":"e8bd044e7cb02d2dd4669ea117ee51f1d9389774bd41de18991850e69e797e93"}
Sep 17 18:06:13.338031 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:13.337877    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8bd044e7cb02d2dd4669ea117ee51f1d9389774bd41de18991850e69e797e93"
Sep 17 18:06:25.485456 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.485425    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j"]
Sep 17 18:06:25.485904 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.485858    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f612f22-acc0-41bc-99e0-6db0a1eb887e" containerName="s3-init"
Sep 17 18:06:25.485904 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.485875    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f612f22-acc0-41bc-99e0-6db0a1eb887e" containerName="s3-init"
Sep 17 18:06:25.485904 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.485886    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f612f22-acc0-41bc-99e0-6db0a1eb887e" containerName="download-hf-model"
Sep 17 18:06:25.485904 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.485893    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f612f22-acc0-41bc-99e0-6db0a1eb887e" containerName="download-hf-model"
Sep 17 18:06:25.486105 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.485994    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f612f22-acc0-41bc-99e0-6db0a1eb887e" containerName="s3-init"
Sep 17 18:06:25.489378 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.489359    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j"
Sep 17 18:06:25.492761 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.492736    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 17 18:06:25.492899 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.492802    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:06:25.493304 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.493263    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 18:06:25.493424 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.493398    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:06:25.493534 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.493517    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w6q8k\""
Sep 17 18:06:25.493619 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.493605    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:06:25.498289 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.498273    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:06:25.503274 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.503253    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-cch4j\" (UID: \"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j"
Sep 17 18:06:25.503372 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.503295    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-cch4j\" (UID: \"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j"
Sep 17 18:06:25.503437 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.503383    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9fpc\" (UniqueName: \"kubernetes.io/projected/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8-kube-api-access-f9fpc\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-cch4j\" (UID: \"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j"
Sep 17 18:06:25.503437 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.503427    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-cch4j\" (UID: \"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j"
Sep 17 18:06:25.503547 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.503488    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-cch4j\" (UID: \"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j"
Sep 17 18:06:25.515169 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.515128    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j"]
Sep 17 18:06:25.577108 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.577083    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl"]
Sep 17 18:06:25.580865 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.580847    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl"
Sep 17 18:06:25.583670 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.583651    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 18:06:25.599374 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.599342    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl"]
Sep 17 18:06:25.604043 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.604018    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-sb9nl\" (UID: \"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl"
Sep 17 18:06:25.604177 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.604060    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-sb9nl\" (UID: \"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl"
Sep 17 18:06:25.604177 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.604138    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9fpc\" (UniqueName: \"kubernetes.io/projected/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8-kube-api-access-f9fpc\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-cch4j\" (UID: \"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j"
Sep 17 18:06:25.604311 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.604199    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7zv9\" (UniqueName: \"kubernetes.io/projected/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-kube-api-access-z7zv9\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-sb9nl\" (UID: \"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl"
Sep 17 18:06:25.604311 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.604241    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-cch4j\" (UID: \"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j"
Sep 17 18:06:25.604311 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.604296    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-cch4j\" (UID: \"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j"
Sep 17 18:06:25.604478 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.604323    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-cch4j\" (UID: \"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j"
Sep 17 18:06:25.604478 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.604359    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-sb9nl\" (UID: \"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl"
Sep 17 18:06:25.604478 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.604386    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-cch4j\" (UID: \"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j"
Sep 17 18:06:25.604478 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.604411    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-sb9nl\" (UID: \"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl"
Sep 17 18:06:25.604806 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.604782    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-cch4j\" (UID: \"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j"
Sep 17 18:06:25.605391 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.605370    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-cch4j\" (UID: \"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j"
Sep 17 18:06:25.605520 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.605430    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-cch4j\" (UID: \"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j"
Sep 17 18:06:25.608302 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.608272    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-cch4j\" (UID: \"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j"
Sep 17 18:06:25.631564 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.631541    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9fpc\" (UniqueName: \"kubernetes.io/projected/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8-kube-api-access-f9fpc\") pod \"isvc-canary-rollback-v2-predictor-65f95c5784-cch4j\" (UID: \"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j"
Sep 17 18:06:25.705061 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.705025    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-sb9nl\" (UID: \"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl"
Sep 17 18:06:25.705227 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.705071    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-sb9nl\" (UID: \"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl"
Sep 17 18:06:25.705227 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.705120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-sb9nl\" (UID: \"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl"
Sep 17 18:06:25.705227 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.705166    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-sb9nl\" (UID: \"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl"
Sep 17 18:06:25.705227 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.705215    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7zv9\" (UniqueName: \"kubernetes.io/projected/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-kube-api-access-z7zv9\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-sb9nl\" (UID: \"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl"
Sep 17 18:06:25.705494 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:06:25.705299    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 18:06:25.705494 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:06:25.705404    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-proxy-tls podName:00b3ff27-6d27-4b65-bdd5-a3b3abe63dad nodeName:}" failed. No retries permitted until 2026-09-17 18:06:26.205382798 +0000 UTC m=+568.305846596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-proxy-tls") pod "isvc-canary-rollback-predictor-7df7cf66bf-sb9nl" (UID: "00b3ff27-6d27-4b65-bdd5-a3b3abe63dad") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 18:06:25.705611 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.705537    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-sb9nl\" (UID: \"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl"
Sep 17 18:06:25.705787 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.705766    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-sb9nl\" (UID: \"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl"
Sep 17 18:06:25.705897 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.705876    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-sb9nl\" (UID: \"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl"
Sep 17 18:06:25.718831 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.718801    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7zv9\" (UniqueName: \"kubernetes.io/projected/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-kube-api-access-z7zv9\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-sb9nl\" (UID: \"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl"
Sep 17 18:06:25.801785 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:25.801693    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j"
Sep 17 18:06:26.090211 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:26.090104    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j"]
Sep 17 18:06:26.093866 ip-10-0-134-209 kubenswrapper[2581]: W0917 18:06:26.093837    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b5ad9a2_ef90_452f_86ec_bf4e2a80e1b8.slice/crio-b0cd0ad4190ce705bd7e20ab6cd8114ad1d201c178eb355e1c04be7830ae549b WatchSource:0}: Error finding container b0cd0ad4190ce705bd7e20ab6cd8114ad1d201c178eb355e1c04be7830ae549b: Status 404 returned error can't find the container with id b0cd0ad4190ce705bd7e20ab6cd8114ad1d201c178eb355e1c04be7830ae549b
Sep 17 18:06:26.208457 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:26.208420    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-sb9nl\" (UID: \"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl"
Sep 17 18:06:26.210702 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:26.210681    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-sb9nl\" (UID: \"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl"
Sep 17 18:06:26.390240 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:26.390193    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j" event={"ID":"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8","Type":"ContainerStarted","Data":"a05f261f4768445b5272711204f39c7574c4a6f180d24da46c6eeec1af5135d9"}
Sep 17 18:06:26.390419 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:26.390279    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j" event={"ID":"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8","Type":"ContainerStarted","Data":"b0cd0ad4190ce705bd7e20ab6cd8114ad1d201c178eb355e1c04be7830ae549b"}
Sep 17 18:06:26.490274 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:26.490243    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl"
Sep 17 18:06:26.769525 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:26.769453    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl"]
Sep 17 18:06:26.773576 ip-10-0-134-209 kubenswrapper[2581]: W0917 18:06:26.773546    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b3ff27_6d27_4b65_bdd5_a3b3abe63dad.slice/crio-903f9018cc9c0f43dc45907b5e8f706240fec5c92647b90f91564dea69e8fd69 WatchSource:0}: Error finding container 903f9018cc9c0f43dc45907b5e8f706240fec5c92647b90f91564dea69e8fd69: Status 404 returned error can't find the container with id 903f9018cc9c0f43dc45907b5e8f706240fec5c92647b90f91564dea69e8fd69
Sep 17 18:06:27.396831 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:27.396802    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl" event={"ID":"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad","Type":"ContainerStarted","Data":"571f867adb857c06762021139e2de3c07147894d0ec7b9d6b6b053bdf1303337"}
Sep 17 18:06:27.397010 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:27.396858    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl" event={"ID":"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad","Type":"ContainerStarted","Data":"903f9018cc9c0f43dc45907b5e8f706240fec5c92647b90f91564dea69e8fd69"}
Sep 17 18:06:30.410348 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:30.410316    2581 generic.go:356] "Generic (PLEG): container finished" podID="1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8" containerID="a05f261f4768445b5272711204f39c7574c4a6f180d24da46c6eeec1af5135d9" exitCode=0
Sep 17 18:06:30.410731 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:30.410385    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j" event={"ID":"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8","Type":"ContainerDied","Data":"a05f261f4768445b5272711204f39c7574c4a6f180d24da46c6eeec1af5135d9"}
Sep 17 18:06:31.420838 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:31.420792    2581 generic.go:356] "Generic (PLEG): container finished" podID="00b3ff27-6d27-4b65-bdd5-a3b3abe63dad" containerID="571f867adb857c06762021139e2de3c07147894d0ec7b9d6b6b053bdf1303337" exitCode=0
Sep 17 18:06:31.421317 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:31.420925    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl" event={"ID":"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad","Type":"ContainerDied","Data":"571f867adb857c06762021139e2de3c07147894d0ec7b9d6b6b053bdf1303337"}
Sep 17 18:06:47.524855 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:47.524809    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j" event={"ID":"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8","Type":"ContainerStarted","Data":"277caf80d0ececf0762e26f016e8c4bc2afc13ecb7a8f03b97bc4b1ea6d714be"}
Sep 17 18:06:47.527273 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:47.527241    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl" event={"ID":"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad","Type":"ContainerStarted","Data":"97122bc602de40580a3006fd0011be023c7e9970e4186d2ee8efe7a0c71af622"}
Sep 17 18:06:50.544828 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:50.544789    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j" event={"ID":"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8","Type":"ContainerStarted","Data":"b98a91a5352e0d41b6d2ddcc32ecf19fe9ec60f5abe575bacc36b396646a62f8"}
Sep 17 18:06:50.545318 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:50.544908    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j"
Sep 17 18:06:50.546905 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:50.546874    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl" event={"ID":"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad","Type":"ContainerStarted","Data":"0b4f5ff722c81c5e406753b49f307e642bd141b93858361f7e05f11b4a3d8004"}
Sep 17 18:06:50.547074 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:50.547061    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl"
Sep 17 18:06:50.547125 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:50.547088    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl"
Sep 17 18:06:50.548549 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:50.548520    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl" podUID="00b3ff27-6d27-4b65-bdd5-a3b3abe63dad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 18:06:50.566574 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:50.566534    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j" podStartSLOduration=2.500889992 podStartE2EDuration="25.566520861s" totalImagesPullingTime="19.171289217s" totalInitContainerRuntime="3.894341652s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:06:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:06:50.56463633 +0000 UTC m=+592.665100149" watchObservedRunningTime="2026-09-17 18:06:50.566520861 +0000 UTC m=+592.666984680"
Sep 17 18:06:50.586567 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:50.586528    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl" podStartSLOduration=3.582285137 podStartE2EDuration="25.586515487s" totalImagesPullingTime="18.248803996s" totalInitContainerRuntime="3.755426354s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:06:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:06:50.584960932 +0000 UTC m=+592.685424754" watchObservedRunningTime="2026-09-17 18:06:50.586515487 +0000 UTC m=+592.686979307"
Sep 17 18:06:51.551036 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:51.550997    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j"
Sep 17 18:06:51.551487 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:51.551029    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl" podUID="00b3ff27-6d27-4b65-bdd5-a3b3abe63dad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 18:06:51.552294 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:51.552269    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j" podUID="1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 18:06:52.555283 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:52.555246    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j" podUID="1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 18:06:56.555582 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:56.555550    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl"
Sep 17 18:06:56.556031 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:56.556007    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl" podUID="00b3ff27-6d27-4b65-bdd5-a3b3abe63dad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 18:06:57.559972 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:57.559942    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j"
Sep 17 18:06:57.560485 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:57.560460    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j" podUID="1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 18:06:59.937357 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:59.937332    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm8hx_913ce1e9-9fb6-4049-a06d-9ebc69d3e528/ovn-acl-logging/0.log"
Sep 17 18:06:59.937762 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:06:59.937618    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm8hx_913ce1e9-9fb6-4049-a06d-9ebc69d3e528/ovn-acl-logging/0.log"
Sep 17 18:07:06.556313 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:07:06.556272    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl" podUID="00b3ff27-6d27-4b65-bdd5-a3b3abe63dad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 18:07:07.560758 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:07:07.560723    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j" podUID="1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 18:07:16.556567 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:07:16.556467    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl" podUID="00b3ff27-6d27-4b65-bdd5-a3b3abe63dad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 18:07:17.561181 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:07:17.561125    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j" podUID="1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 18:07:26.556253 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:07:26.556212    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl" podUID="00b3ff27-6d27-4b65-bdd5-a3b3abe63dad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 18:07:27.560725 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:07:27.560686    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j" podUID="1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 18:07:36.556455 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:07:36.556414    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl" podUID="00b3ff27-6d27-4b65-bdd5-a3b3abe63dad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 18:07:37.560729 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:07:37.560692    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j" podUID="1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 18:07:46.557045 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:07:46.557006    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl" podUID="00b3ff27-6d27-4b65-bdd5-a3b3abe63dad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 18:07:47.560575 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:07:47.560536    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j" podUID="1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 18:07:56.556532 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:07:56.556488    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl" podUID="00b3ff27-6d27-4b65-bdd5-a3b3abe63dad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 18:07:57.561399 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:07:57.561357    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j" podUID="1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 18:08:01.044349 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:01.044312    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j"
Sep 17 18:08:01.261474 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:01.261444    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl"]
Sep 17 18:08:01.261729 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:01.261672    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl" podUID="00b3ff27-6d27-4b65-bdd5-a3b3abe63dad" containerName="kserve-container" containerID="cri-o://97122bc602de40580a3006fd0011be023c7e9970e4186d2ee8efe7a0c71af622" gracePeriod=30
Sep 17 18:08:01.261873 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:01.261734    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl" podUID="00b3ff27-6d27-4b65-bdd5-a3b3abe63dad" containerName="kube-rbac-proxy" containerID="cri-o://0b4f5ff722c81c5e406753b49f307e642bd141b93858361f7e05f11b4a3d8004" gracePeriod=30
Sep 17 18:08:01.262536 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:01.262256    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl"
Sep 17 18:08:01.551581 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:01.551545    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl" podUID="00b3ff27-6d27-4b65-bdd5-a3b3abe63dad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Sep 17 18:08:01.843192 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:01.843096    2581 generic.go:356] "Generic (PLEG): container finished" podID="00b3ff27-6d27-4b65-bdd5-a3b3abe63dad" containerID="0b4f5ff722c81c5e406753b49f307e642bd141b93858361f7e05f11b4a3d8004" exitCode=2
Sep 17 18:08:01.843192 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:01.843175    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl" event={"ID":"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad","Type":"ContainerDied","Data":"0b4f5ff722c81c5e406753b49f307e642bd141b93858361f7e05f11b4a3d8004"}
Sep 17 18:08:05.503860 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.503835    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl"
Sep 17 18:08:05.646411 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.646379    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-proxy-tls\") pod \"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad\" (UID: \"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad\") "
Sep 17 18:08:05.646569 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.646436    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7zv9\" (UniqueName: \"kubernetes.io/projected/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-kube-api-access-z7zv9\") pod \"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad\" (UID: \"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad\") "
Sep 17 18:08:05.646569 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.646472    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-kube-rbac-proxy-sar-config\") pod \"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad\" (UID: \"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad\") "
Sep 17 18:08:05.646569 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.646508    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-kserve-provision-location\") pod \"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad\" (UID: \"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad\") "
Sep 17 18:08:05.646569 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.646530    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-cabundle-cert\") pod \"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad\" (UID: \"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad\") "
Sep 17 18:08:05.646881 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.646817    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "00b3ff27-6d27-4b65-bdd5-a3b3abe63dad" (UID: "00b3ff27-6d27-4b65-bdd5-a3b3abe63dad"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:08:05.646979 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.646875    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "00b3ff27-6d27-4b65-bdd5-a3b3abe63dad" (UID: "00b3ff27-6d27-4b65-bdd5-a3b3abe63dad"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:08:05.646979 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.646967    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "00b3ff27-6d27-4b65-bdd5-a3b3abe63dad" (UID: "00b3ff27-6d27-4b65-bdd5-a3b3abe63dad"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:08:05.648764 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.648741    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "00b3ff27-6d27-4b65-bdd5-a3b3abe63dad" (UID: "00b3ff27-6d27-4b65-bdd5-a3b3abe63dad"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:08:05.648835 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.648767    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-kube-api-access-z7zv9" (OuterVolumeSpecName: "kube-api-access-z7zv9") pod "00b3ff27-6d27-4b65-bdd5-a3b3abe63dad" (UID: "00b3ff27-6d27-4b65-bdd5-a3b3abe63dad"). InnerVolumeSpecName "kube-api-access-z7zv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:08:05.748012 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.747986    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-kserve-provision-location\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:08:05.748012 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.748010    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-cabundle-cert\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:08:05.748180 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.748019    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-proxy-tls\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:08:05.748180 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.748028    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7zv9\" (UniqueName: \"kubernetes.io/projected/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-kube-api-access-z7zv9\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:08:05.748180 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.748038    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:08:05.861783 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.861750    2581 generic.go:356] "Generic (PLEG): container finished" podID="00b3ff27-6d27-4b65-bdd5-a3b3abe63dad" containerID="97122bc602de40580a3006fd0011be023c7e9970e4186d2ee8efe7a0c71af622" exitCode=0
Sep 17 18:08:05.861874 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.861813    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl"
Sep 17 18:08:05.861874 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.861833    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl" event={"ID":"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad","Type":"ContainerDied","Data":"97122bc602de40580a3006fd0011be023c7e9970e4186d2ee8efe7a0c71af622"}
Sep 17 18:08:05.861958 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.861904    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl" event={"ID":"00b3ff27-6d27-4b65-bdd5-a3b3abe63dad","Type":"ContainerDied","Data":"903f9018cc9c0f43dc45907b5e8f706240fec5c92647b90f91564dea69e8fd69"}
Sep 17 18:08:05.861958 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.861932    2581 scope.go:117] "RemoveContainer" containerID="0b4f5ff722c81c5e406753b49f307e642bd141b93858361f7e05f11b4a3d8004"
Sep 17 18:08:05.871532 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.871507    2581 scope.go:117] "RemoveContainer" containerID="97122bc602de40580a3006fd0011be023c7e9970e4186d2ee8efe7a0c71af622"
Sep 17 18:08:05.881696 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.881675    2581 scope.go:117] "RemoveContainer" containerID="571f867adb857c06762021139e2de3c07147894d0ec7b9d6b6b053bdf1303337"
Sep 17 18:08:05.888847 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.888831    2581 scope.go:117] "RemoveContainer" containerID="0b4f5ff722c81c5e406753b49f307e642bd141b93858361f7e05f11b4a3d8004"
Sep 17 18:08:05.889166 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:08:05.889124    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b4f5ff722c81c5e406753b49f307e642bd141b93858361f7e05f11b4a3d8004\": container with ID starting with 0b4f5ff722c81c5e406753b49f307e642bd141b93858361f7e05f11b4a3d8004 not found: ID does not exist" containerID="0b4f5ff722c81c5e406753b49f307e642bd141b93858361f7e05f11b4a3d8004"
Sep 17 18:08:05.889259 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.889178    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b4f5ff722c81c5e406753b49f307e642bd141b93858361f7e05f11b4a3d8004"} err="failed to get container status \"0b4f5ff722c81c5e406753b49f307e642bd141b93858361f7e05f11b4a3d8004\": rpc error: code = NotFound desc = could not find container \"0b4f5ff722c81c5e406753b49f307e642bd141b93858361f7e05f11b4a3d8004\": container with ID starting with 0b4f5ff722c81c5e406753b49f307e642bd141b93858361f7e05f11b4a3d8004 not found: ID does not exist"
Sep 17 18:08:05.889259 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.889203    2581 scope.go:117] "RemoveContainer" containerID="97122bc602de40580a3006fd0011be023c7e9970e4186d2ee8efe7a0c71af622"
Sep 17 18:08:05.889533 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:08:05.889514    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97122bc602de40580a3006fd0011be023c7e9970e4186d2ee8efe7a0c71af622\": container with ID starting with 97122bc602de40580a3006fd0011be023c7e9970e4186d2ee8efe7a0c71af622 not found: ID does not exist" containerID="97122bc602de40580a3006fd0011be023c7e9970e4186d2ee8efe7a0c71af622"
Sep 17 18:08:05.889573 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.889538    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97122bc602de40580a3006fd0011be023c7e9970e4186d2ee8efe7a0c71af622"} err="failed to get container status \"97122bc602de40580a3006fd0011be023c7e9970e4186d2ee8efe7a0c71af622\": rpc error: code = NotFound desc = could not find container \"97122bc602de40580a3006fd0011be023c7e9970e4186d2ee8efe7a0c71af622\": container with ID starting with 97122bc602de40580a3006fd0011be023c7e9970e4186d2ee8efe7a0c71af622 not found: ID does not exist"
Sep 17 18:08:05.889573 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.889554    2581 scope.go:117] "RemoveContainer" containerID="571f867adb857c06762021139e2de3c07147894d0ec7b9d6b6b053bdf1303337"
Sep 17 18:08:05.889851 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:08:05.889829    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"571f867adb857c06762021139e2de3c07147894d0ec7b9d6b6b053bdf1303337\": container with ID starting with 571f867adb857c06762021139e2de3c07147894d0ec7b9d6b6b053bdf1303337 not found: ID does not exist" containerID="571f867adb857c06762021139e2de3c07147894d0ec7b9d6b6b053bdf1303337"
Sep 17 18:08:05.889895 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.889855    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"571f867adb857c06762021139e2de3c07147894d0ec7b9d6b6b053bdf1303337"} err="failed to get container status \"571f867adb857c06762021139e2de3c07147894d0ec7b9d6b6b053bdf1303337\": rpc error: code = NotFound desc = could not find container \"571f867adb857c06762021139e2de3c07147894d0ec7b9d6b6b053bdf1303337\": container with ID starting with 571f867adb857c06762021139e2de3c07147894d0ec7b9d6b6b053bdf1303337 not found: ID does not exist"
Sep 17 18:08:05.892282 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.892262    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl"]
Sep 17 18:08:05.895393 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:05.895374    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-sb9nl"]
Sep 17 18:08:06.048568 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:06.048511    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00b3ff27-6d27-4b65-bdd5-a3b3abe63dad" path="/var/lib/kubelet/pods/00b3ff27-6d27-4b65-bdd5-a3b3abe63dad/volumes"
Sep 17 18:08:15.192007 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.191973    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j"]
Sep 17 18:08:15.192390 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.192232    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j" podUID="1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8" containerName="kserve-container" containerID="cri-o://277caf80d0ececf0762e26f016e8c4bc2afc13ecb7a8f03b97bc4b1ea6d714be" gracePeriod=30
Sep 17 18:08:15.192390 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.192300    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j" podUID="1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8" containerName="kube-rbac-proxy" containerID="cri-o://b98a91a5352e0d41b6d2ddcc32ecf19fe9ec60f5abe575bacc36b396646a62f8" gracePeriod=30
Sep 17 18:08:15.259213 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.259180    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv"]
Sep 17 18:08:15.259492 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.259479    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00b3ff27-6d27-4b65-bdd5-a3b3abe63dad" containerName="kube-rbac-proxy"
Sep 17 18:08:15.259538 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.259494    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="00b3ff27-6d27-4b65-bdd5-a3b3abe63dad" containerName="kube-rbac-proxy"
Sep 17 18:08:15.259538 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.259509    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00b3ff27-6d27-4b65-bdd5-a3b3abe63dad" containerName="storage-initializer"
Sep 17 18:08:15.259538 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.259514    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="00b3ff27-6d27-4b65-bdd5-a3b3abe63dad" containerName="storage-initializer"
Sep 17 18:08:15.259538 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.259528    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00b3ff27-6d27-4b65-bdd5-a3b3abe63dad" containerName="kserve-container"
Sep 17 18:08:15.259538 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.259534    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="00b3ff27-6d27-4b65-bdd5-a3b3abe63dad" containerName="kserve-container"
Sep 17 18:08:15.259682 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.259583    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="00b3ff27-6d27-4b65-bdd5-a3b3abe63dad" containerName="kserve-container"
Sep 17 18:08:15.259682 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.259594    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="00b3ff27-6d27-4b65-bdd5-a3b3abe63dad" containerName="kube-rbac-proxy"
Sep 17 18:08:15.261889 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.261871    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv"
Sep 17 18:08:15.263965 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.263948    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 18:08:15.275247 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.275227    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv"]
Sep 17 18:08:15.428448 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.428387    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a5b7de5-725a-4768-9c72-2e7d04e033c0-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-nrzjv\" (UID: \"2a5b7de5-725a-4768-9c72-2e7d04e033c0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv"
Sep 17 18:08:15.428646 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.428483    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a5b7de5-725a-4768-9c72-2e7d04e033c0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-nrzjv\" (UID: \"2a5b7de5-725a-4768-9c72-2e7d04e033c0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv"
Sep 17 18:08:15.428646 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.428523    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a5b7de5-725a-4768-9c72-2e7d04e033c0-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-nrzjv\" (UID: \"2a5b7de5-725a-4768-9c72-2e7d04e033c0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv"
Sep 17 18:08:15.428646 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.428567    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a5b7de5-725a-4768-9c72-2e7d04e033c0-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-nrzjv\" (UID: \"2a5b7de5-725a-4768-9c72-2e7d04e033c0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv"
Sep 17 18:08:15.428646 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.428590    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfs5q\" (UniqueName: \"kubernetes.io/projected/2a5b7de5-725a-4768-9c72-2e7d04e033c0-kube-api-access-xfs5q\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-nrzjv\" (UID: \"2a5b7de5-725a-4768-9c72-2e7d04e033c0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv"
Sep 17 18:08:15.529350 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.529268    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a5b7de5-725a-4768-9c72-2e7d04e033c0-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-nrzjv\" (UID: \"2a5b7de5-725a-4768-9c72-2e7d04e033c0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv"
Sep 17 18:08:15.529350 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.529312    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a5b7de5-725a-4768-9c72-2e7d04e033c0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-nrzjv\" (UID: \"2a5b7de5-725a-4768-9c72-2e7d04e033c0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv"
Sep 17 18:08:15.529556 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.529352    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a5b7de5-725a-4768-9c72-2e7d04e033c0-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-nrzjv\" (UID: \"2a5b7de5-725a-4768-9c72-2e7d04e033c0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv"
Sep 17 18:08:15.529556 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.529391    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a5b7de5-725a-4768-9c72-2e7d04e033c0-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-nrzjv\" (UID: \"2a5b7de5-725a-4768-9c72-2e7d04e033c0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv"
Sep 17 18:08:15.529556 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.529414    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfs5q\" (UniqueName: \"kubernetes.io/projected/2a5b7de5-725a-4768-9c72-2e7d04e033c0-kube-api-access-xfs5q\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-nrzjv\" (UID: \"2a5b7de5-725a-4768-9c72-2e7d04e033c0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv"
Sep 17 18:08:15.529820 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.529798    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a5b7de5-725a-4768-9c72-2e7d04e033c0-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-nrzjv\" (UID: \"2a5b7de5-725a-4768-9c72-2e7d04e033c0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv"
Sep 17 18:08:15.530027 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.530008    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a5b7de5-725a-4768-9c72-2e7d04e033c0-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-nrzjv\" (UID: \"2a5b7de5-725a-4768-9c72-2e7d04e033c0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv"
Sep 17 18:08:15.530076 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.530060    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a5b7de5-725a-4768-9c72-2e7d04e033c0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-nrzjv\" (UID: \"2a5b7de5-725a-4768-9c72-2e7d04e033c0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv"
Sep 17 18:08:15.532006 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.531989    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a5b7de5-725a-4768-9c72-2e7d04e033c0-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-nrzjv\" (UID: \"2a5b7de5-725a-4768-9c72-2e7d04e033c0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv"
Sep 17 18:08:15.538258 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.538237    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfs5q\" (UniqueName: \"kubernetes.io/projected/2a5b7de5-725a-4768-9c72-2e7d04e033c0-kube-api-access-xfs5q\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-nrzjv\" (UID: \"2a5b7de5-725a-4768-9c72-2e7d04e033c0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv"
Sep 17 18:08:15.572052 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.572025    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv"
Sep 17 18:08:15.868365 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.868334    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv"]
Sep 17 18:08:15.871292 ip-10-0-134-209 kubenswrapper[2581]: W0917 18:08:15.871270    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5b7de5_725a_4768_9c72_2e7d04e033c0.slice/crio-d654b1cf8749bdf1452fdbe89100d063a76f3493bc85ae8343242e0135c4a5ed WatchSource:0}: Error finding container d654b1cf8749bdf1452fdbe89100d063a76f3493bc85ae8343242e0135c4a5ed: Status 404 returned error can't find the container with id d654b1cf8749bdf1452fdbe89100d063a76f3493bc85ae8343242e0135c4a5ed
Sep 17 18:08:15.904006 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.903986    2581 generic.go:356] "Generic (PLEG): container finished" podID="1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8" containerID="b98a91a5352e0d41b6d2ddcc32ecf19fe9ec60f5abe575bacc36b396646a62f8" exitCode=2
Sep 17 18:08:15.904115 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.904064    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j" event={"ID":"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8","Type":"ContainerDied","Data":"b98a91a5352e0d41b6d2ddcc32ecf19fe9ec60f5abe575bacc36b396646a62f8"}
Sep 17 18:08:15.905241 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:15.905223    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv" event={"ID":"2a5b7de5-725a-4768-9c72-2e7d04e033c0","Type":"ContainerStarted","Data":"d654b1cf8749bdf1452fdbe89100d063a76f3493bc85ae8343242e0135c4a5ed"}
Sep 17 18:08:16.910641 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:16.910605    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv" event={"ID":"2a5b7de5-725a-4768-9c72-2e7d04e033c0","Type":"ContainerStarted","Data":"7ba513683e56e7246a7e78577c124b7561f7f9377d4960545ce9253b6b21126b"}
Sep 17 18:08:17.555516 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:17.555480    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j" podUID="1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 17 18:08:19.739420 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.739398    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j"
Sep 17 18:08:19.764288 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.764264    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8-cabundle-cert\") pod \"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8\" (UID: \"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8\") "
Sep 17 18:08:19.764424 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.764305    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8-kube-rbac-proxy-sar-config\") pod \"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8\" (UID: \"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8\") "
Sep 17 18:08:19.764424 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.764336    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8-kserve-provision-location\") pod \"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8\" (UID: \"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8\") "
Sep 17 18:08:19.764424 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.764385    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9fpc\" (UniqueName: \"kubernetes.io/projected/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8-kube-api-access-f9fpc\") pod \"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8\" (UID: \"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8\") "
Sep 17 18:08:19.764600 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.764428    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8-proxy-tls\") pod \"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8\" (UID: \"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8\") "
Sep 17 18:08:19.764672 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.764645    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8" (UID: "1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:08:19.764730 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.764657    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8" (UID: "1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:08:19.764730 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.764688    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8" (UID: "1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:08:19.766764 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.766732    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8" (UID: "1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:08:19.766764 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.766746    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8-kube-api-access-f9fpc" (OuterVolumeSpecName: "kube-api-access-f9fpc") pod "1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8" (UID: "1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8"). InnerVolumeSpecName "kube-api-access-f9fpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:08:19.865569 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.865536    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8-cabundle-cert\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:08:19.865569 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.865569    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:08:19.865759 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.865580    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8-kserve-provision-location\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:08:19.865759 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.865589    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9fpc\" (UniqueName: \"kubernetes.io/projected/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8-kube-api-access-f9fpc\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:08:19.865759 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.865598    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8-proxy-tls\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:08:19.924441 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.924415    2581 generic.go:356] "Generic (PLEG): container finished" podID="2a5b7de5-725a-4768-9c72-2e7d04e033c0" containerID="7ba513683e56e7246a7e78577c124b7561f7f9377d4960545ce9253b6b21126b" exitCode=0
Sep 17 18:08:19.924561 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.924492    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv" event={"ID":"2a5b7de5-725a-4768-9c72-2e7d04e033c0","Type":"ContainerDied","Data":"7ba513683e56e7246a7e78577c124b7561f7f9377d4960545ce9253b6b21126b"}
Sep 17 18:08:19.926496 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.926468    2581 generic.go:356] "Generic (PLEG): container finished" podID="1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8" containerID="277caf80d0ececf0762e26f016e8c4bc2afc13ecb7a8f03b97bc4b1ea6d714be" exitCode=0
Sep 17 18:08:19.926599 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.926516    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j" event={"ID":"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8","Type":"ContainerDied","Data":"277caf80d0ececf0762e26f016e8c4bc2afc13ecb7a8f03b97bc4b1ea6d714be"}
Sep 17 18:08:19.926599 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.926522    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j"
Sep 17 18:08:19.926599 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.926571    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j" event={"ID":"1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8","Type":"ContainerDied","Data":"b0cd0ad4190ce705bd7e20ab6cd8114ad1d201c178eb355e1c04be7830ae549b"}
Sep 17 18:08:19.926701 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.926614    2581 scope.go:117] "RemoveContainer" containerID="b98a91a5352e0d41b6d2ddcc32ecf19fe9ec60f5abe575bacc36b396646a62f8"
Sep 17 18:08:19.938056 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.938038    2581 scope.go:117] "RemoveContainer" containerID="277caf80d0ececf0762e26f016e8c4bc2afc13ecb7a8f03b97bc4b1ea6d714be"
Sep 17 18:08:19.946429 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.946406    2581 scope.go:117] "RemoveContainer" containerID="a05f261f4768445b5272711204f39c7574c4a6f180d24da46c6eeec1af5135d9"
Sep 17 18:08:19.957125 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.957103    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j"]
Sep 17 18:08:19.958504 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.958486    2581 scope.go:117] "RemoveContainer" containerID="b98a91a5352e0d41b6d2ddcc32ecf19fe9ec60f5abe575bacc36b396646a62f8"
Sep 17 18:08:19.958791 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:08:19.958773    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b98a91a5352e0d41b6d2ddcc32ecf19fe9ec60f5abe575bacc36b396646a62f8\": container with ID starting with b98a91a5352e0d41b6d2ddcc32ecf19fe9ec60f5abe575bacc36b396646a62f8 not found: ID does not exist" containerID="b98a91a5352e0d41b6d2ddcc32ecf19fe9ec60f5abe575bacc36b396646a62f8"
Sep 17 18:08:19.958860 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.958798    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b98a91a5352e0d41b6d2ddcc32ecf19fe9ec60f5abe575bacc36b396646a62f8"} err="failed to get container status \"b98a91a5352e0d41b6d2ddcc32ecf19fe9ec60f5abe575bacc36b396646a62f8\": rpc error: code = NotFound desc = could not find container \"b98a91a5352e0d41b6d2ddcc32ecf19fe9ec60f5abe575bacc36b396646a62f8\": container with ID starting with b98a91a5352e0d41b6d2ddcc32ecf19fe9ec60f5abe575bacc36b396646a62f8 not found: ID does not exist"
Sep 17 18:08:19.958860 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.958817    2581 scope.go:117] "RemoveContainer" containerID="277caf80d0ececf0762e26f016e8c4bc2afc13ecb7a8f03b97bc4b1ea6d714be"
Sep 17 18:08:19.959183 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:08:19.959129    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"277caf80d0ececf0762e26f016e8c4bc2afc13ecb7a8f03b97bc4b1ea6d714be\": container with ID starting with 277caf80d0ececf0762e26f016e8c4bc2afc13ecb7a8f03b97bc4b1ea6d714be not found: ID does not exist" containerID="277caf80d0ececf0762e26f016e8c4bc2afc13ecb7a8f03b97bc4b1ea6d714be"
Sep 17 18:08:19.959275 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.959193    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"277caf80d0ececf0762e26f016e8c4bc2afc13ecb7a8f03b97bc4b1ea6d714be"} err="failed to get container status \"277caf80d0ececf0762e26f016e8c4bc2afc13ecb7a8f03b97bc4b1ea6d714be\": rpc error: code = NotFound desc = could not find container \"277caf80d0ececf0762e26f016e8c4bc2afc13ecb7a8f03b97bc4b1ea6d714be\": container with ID starting with 277caf80d0ececf0762e26f016e8c4bc2afc13ecb7a8f03b97bc4b1ea6d714be not found: ID does not exist"
Sep 17 18:08:19.959275 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.959215    2581 scope.go:117] "RemoveContainer" containerID="a05f261f4768445b5272711204f39c7574c4a6f180d24da46c6eeec1af5135d9"
Sep 17 18:08:19.959542 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:08:19.959517    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a05f261f4768445b5272711204f39c7574c4a6f180d24da46c6eeec1af5135d9\": container with ID starting with a05f261f4768445b5272711204f39c7574c4a6f180d24da46c6eeec1af5135d9 not found: ID does not exist" containerID="a05f261f4768445b5272711204f39c7574c4a6f180d24da46c6eeec1af5135d9"
Sep 17 18:08:19.959624 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.959549    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a05f261f4768445b5272711204f39c7574c4a6f180d24da46c6eeec1af5135d9"} err="failed to get container status \"a05f261f4768445b5272711204f39c7574c4a6f180d24da46c6eeec1af5135d9\": rpc error: code = NotFound desc = could not find container \"a05f261f4768445b5272711204f39c7574c4a6f180d24da46c6eeec1af5135d9\": container with ID starting with a05f261f4768445b5272711204f39c7574c4a6f180d24da46c6eeec1af5135d9 not found: ID does not exist"
Sep 17 18:08:19.961682 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:19.961664    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-65f95c5784-cch4j"]
Sep 17 18:08:20.049397 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.049375    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8" path="/var/lib/kubelet/pods/1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8/volumes"
Sep 17 18:08:20.360397 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.360316    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn"]
Sep 17 18:08:20.360726 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.360708    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8" containerName="storage-initializer"
Sep 17 18:08:20.360814 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.360729    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8" containerName="storage-initializer"
Sep 17 18:08:20.360814 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.360759    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8" containerName="kserve-container"
Sep 17 18:08:20.360814 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.360768    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8" containerName="kserve-container"
Sep 17 18:08:20.360814 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.360782    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8" containerName="kube-rbac-proxy"
Sep 17 18:08:20.360814 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.360791    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8" containerName="kube-rbac-proxy"
Sep 17 18:08:20.361056 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.360866    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8" containerName="kserve-container"
Sep 17 18:08:20.361056 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.360881    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b5ad9a2-ef90-452f-86ec-bf4e2a80e1b8" containerName="kube-rbac-proxy"
Sep 17 18:08:20.363109 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.363088    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn"
Sep 17 18:08:20.365032 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.365014    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-0cba7-predictor-serving-cert\""
Sep 17 18:08:20.365319 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.365296    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-0cba7-kube-rbac-proxy-sar-config\""
Sep 17 18:08:20.373203 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.373180    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn"]
Sep 17 18:08:20.471285 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.471256    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bd93e17-0115-4979-883a-aef312aa68b3-proxy-tls\") pod \"isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn\" (UID: \"2bd93e17-0115-4979-883a-aef312aa68b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn"
Sep 17 18:08:20.471444 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.471330    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bd93e17-0115-4979-883a-aef312aa68b3-cabundle-cert\") pod \"isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn\" (UID: \"2bd93e17-0115-4979-883a-aef312aa68b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn"
Sep 17 18:08:20.471444 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.471364    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bd93e17-0115-4979-883a-aef312aa68b3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn\" (UID: \"2bd93e17-0115-4979-883a-aef312aa68b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn"
Sep 17 18:08:20.471444 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.471440    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8vfl\" (UniqueName: \"kubernetes.io/projected/2bd93e17-0115-4979-883a-aef312aa68b3-kube-api-access-l8vfl\") pod \"isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn\" (UID: \"2bd93e17-0115-4979-883a-aef312aa68b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn"
Sep 17 18:08:20.471650 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.471478    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bd93e17-0115-4979-883a-aef312aa68b3-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn\" (UID: \"2bd93e17-0115-4979-883a-aef312aa68b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn"
Sep 17 18:08:20.511249 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.511204    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n"]
Sep 17 18:08:20.514804 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.514786    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n"
Sep 17 18:08:20.517353 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.517330    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-0cba7-predictor-serving-cert\""
Sep 17 18:08:20.517451 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.517378    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-0cba7-kube-rbac-proxy-sar-config\""
Sep 17 18:08:20.530297 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.530274    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n"]
Sep 17 18:08:20.572179 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.572135    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bd93e17-0115-4979-883a-aef312aa68b3-proxy-tls\") pod \"isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn\" (UID: \"2bd93e17-0115-4979-883a-aef312aa68b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn"
Sep 17 18:08:20.572278 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.572212    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0fa9a63c-679c-4161-b889-674a6a4c198e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n\" (UID: \"0fa9a63c-679c-4161-b889-674a6a4c198e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n"
Sep 17 18:08:20.572278 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:08:20.572237    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-serving-cert: secret "isvc-sklearn-graph-raw-0cba7-predictor-serving-cert" not found
Sep 17 18:08:20.572278 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.572266    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bd93e17-0115-4979-883a-aef312aa68b3-cabundle-cert\") pod \"isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn\" (UID: \"2bd93e17-0115-4979-883a-aef312aa68b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn"
Sep 17 18:08:20.572406 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:08:20.572292    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bd93e17-0115-4979-883a-aef312aa68b3-proxy-tls podName:2bd93e17-0115-4979-883a-aef312aa68b3 nodeName:}" failed. No retries permitted until 2026-09-17 18:08:21.07227395 +0000 UTC m=+683.172737755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2bd93e17-0115-4979-883a-aef312aa68b3-proxy-tls") pod "isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn" (UID: "2bd93e17-0115-4979-883a-aef312aa68b3") : secret "isvc-sklearn-graph-raw-0cba7-predictor-serving-cert" not found
Sep 17 18:08:20.572406 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.572369    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0fa9a63c-679c-4161-b889-674a6a4c198e-cabundle-cert\") pod \"isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n\" (UID: \"0fa9a63c-679c-4161-b889-674a6a4c198e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n"
Sep 17 18:08:20.572530 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.572410    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bd93e17-0115-4979-883a-aef312aa68b3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn\" (UID: \"2bd93e17-0115-4979-883a-aef312aa68b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn"
Sep 17 18:08:20.572530 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.572468    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fa9a63c-679c-4161-b889-674a6a4c198e-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n\" (UID: \"0fa9a63c-679c-4161-b889-674a6a4c198e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n"
Sep 17 18:08:20.572530 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.572515    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmtrs\" (UniqueName: \"kubernetes.io/projected/0fa9a63c-679c-4161-b889-674a6a4c198e-kube-api-access-nmtrs\") pod \"isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n\" (UID: \"0fa9a63c-679c-4161-b889-674a6a4c198e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n"
Sep 17 18:08:20.572707 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.572552    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fa9a63c-679c-4161-b889-674a6a4c198e-proxy-tls\") pod \"isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n\" (UID: \"0fa9a63c-679c-4161-b889-674a6a4c198e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n"
Sep 17 18:08:20.572707 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.572589    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8vfl\" (UniqueName: \"kubernetes.io/projected/2bd93e17-0115-4979-883a-aef312aa68b3-kube-api-access-l8vfl\") pod \"isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn\" (UID: \"2bd93e17-0115-4979-883a-aef312aa68b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn"
Sep 17 18:08:20.572707 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.572677    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bd93e17-0115-4979-883a-aef312aa68b3-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn\" (UID: \"2bd93e17-0115-4979-883a-aef312aa68b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn"
Sep 17 18:08:20.572948 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.572927    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bd93e17-0115-4979-883a-aef312aa68b3-cabundle-cert\") pod \"isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn\" (UID: \"2bd93e17-0115-4979-883a-aef312aa68b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn"
Sep 17 18:08:20.573019 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.572945    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bd93e17-0115-4979-883a-aef312aa68b3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn\" (UID: \"2bd93e17-0115-4979-883a-aef312aa68b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn"
Sep 17 18:08:20.573019 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.573007    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bd93e17-0115-4979-883a-aef312aa68b3-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn\" (UID: \"2bd93e17-0115-4979-883a-aef312aa68b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn"
Sep 17 18:08:20.579954 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.579929    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8vfl\" (UniqueName: \"kubernetes.io/projected/2bd93e17-0115-4979-883a-aef312aa68b3-kube-api-access-l8vfl\") pod \"isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn\" (UID: \"2bd93e17-0115-4979-883a-aef312aa68b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn"
Sep 17 18:08:20.673007 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.672984    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmtrs\" (UniqueName: \"kubernetes.io/projected/0fa9a63c-679c-4161-b889-674a6a4c198e-kube-api-access-nmtrs\") pod \"isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n\" (UID: \"0fa9a63c-679c-4161-b889-674a6a4c198e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n"
Sep 17 18:08:20.673115 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.673014    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fa9a63c-679c-4161-b889-674a6a4c198e-proxy-tls\") pod \"isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n\" (UID: \"0fa9a63c-679c-4161-b889-674a6a4c198e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n"
Sep 17 18:08:20.673115 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.673065    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0fa9a63c-679c-4161-b889-674a6a4c198e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n\" (UID: \"0fa9a63c-679c-4161-b889-674a6a4c198e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n"
Sep 17 18:08:20.673115 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.673097    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0fa9a63c-679c-4161-b889-674a6a4c198e-cabundle-cert\") pod \"isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n\" (UID: \"0fa9a63c-679c-4161-b889-674a6a4c198e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n"
Sep 17 18:08:20.673326 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.673123    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fa9a63c-679c-4161-b889-674a6a4c198e-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n\" (UID: \"0fa9a63c-679c-4161-b889-674a6a4c198e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n"
Sep 17 18:08:20.673326 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:08:20.673169    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-serving-cert: secret "isvc-xgboost-graph-raw-0cba7-predictor-serving-cert" not found
Sep 17 18:08:20.673326 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:08:20.673231    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fa9a63c-679c-4161-b889-674a6a4c198e-proxy-tls podName:0fa9a63c-679c-4161-b889-674a6a4c198e nodeName:}" failed. No retries permitted until 2026-09-17 18:08:21.173213962 +0000 UTC m=+683.273677766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0fa9a63c-679c-4161-b889-674a6a4c198e-proxy-tls") pod "isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n" (UID: "0fa9a63c-679c-4161-b889-674a6a4c198e") : secret "isvc-xgboost-graph-raw-0cba7-predictor-serving-cert" not found
Sep 17 18:08:20.673562 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.673543    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fa9a63c-679c-4161-b889-674a6a4c198e-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n\" (UID: \"0fa9a63c-679c-4161-b889-674a6a4c198e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n"
Sep 17 18:08:20.673765 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.673750    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0fa9a63c-679c-4161-b889-674a6a4c198e-cabundle-cert\") pod \"isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n\" (UID: \"0fa9a63c-679c-4161-b889-674a6a4c198e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n"
Sep 17 18:08:20.673895 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.673876    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0fa9a63c-679c-4161-b889-674a6a4c198e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n\" (UID: \"0fa9a63c-679c-4161-b889-674a6a4c198e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n"
Sep 17 18:08:20.681443 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.681418    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmtrs\" (UniqueName: \"kubernetes.io/projected/0fa9a63c-679c-4161-b889-674a6a4c198e-kube-api-access-nmtrs\") pod \"isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n\" (UID: \"0fa9a63c-679c-4161-b889-674a6a4c198e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n"
Sep 17 18:08:20.937837 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.937757    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv" event={"ID":"2a5b7de5-725a-4768-9c72-2e7d04e033c0","Type":"ContainerStarted","Data":"d4752cd229d82fe4cfd22acd8a8b403405724b6633aa1c79d85838725ed26f99"}
Sep 17 18:08:20.938275 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.937861    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv" event={"ID":"2a5b7de5-725a-4768-9c72-2e7d04e033c0","Type":"ContainerStarted","Data":"b39d7946e30b9d6c37473d412b4a57db3cff1fbce378b2269ea8eaa5202f6fc8"}
Sep 17 18:08:20.938275 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.938068    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv"
Sep 17 18:08:20.958950 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:20.958908    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv" podStartSLOduration=2.060263831 podStartE2EDuration="5.95889562s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.898631789s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:20.957172817 +0000 UTC m=+683.057636628" watchObservedRunningTime="2026-09-17 18:08:20.95889562 +0000 UTC m=+683.059359514"
Sep 17 18:08:21.077076 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:21.077048    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bd93e17-0115-4979-883a-aef312aa68b3-proxy-tls\") pod \"isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn\" (UID: \"2bd93e17-0115-4979-883a-aef312aa68b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn"
Sep 17 18:08:21.079563 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:21.079535    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bd93e17-0115-4979-883a-aef312aa68b3-proxy-tls\") pod \"isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn\" (UID: \"2bd93e17-0115-4979-883a-aef312aa68b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn"
Sep 17 18:08:21.177993 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:21.177966    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fa9a63c-679c-4161-b889-674a6a4c198e-proxy-tls\") pod \"isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n\" (UID: \"0fa9a63c-679c-4161-b889-674a6a4c198e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n"
Sep 17 18:08:21.180349 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:21.180323    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fa9a63c-679c-4161-b889-674a6a4c198e-proxy-tls\") pod \"isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n\" (UID: \"0fa9a63c-679c-4161-b889-674a6a4c198e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n"
Sep 17 18:08:21.275444 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:21.275391    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn"
Sep 17 18:08:21.433738 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:21.433707    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n"
Sep 17 18:08:21.550444 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:21.550415    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn"]
Sep 17 18:08:21.553171 ip-10-0-134-209 kubenswrapper[2581]: W0917 18:08:21.553128    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd93e17_0115_4979_883a_aef312aa68b3.slice/crio-76c038277fc7630a456001dbaf82c77fcb3290fc64a2e2cc6b12374ffb4d3dda WatchSource:0}: Error finding container 76c038277fc7630a456001dbaf82c77fcb3290fc64a2e2cc6b12374ffb4d3dda: Status 404 returned error can't find the container with id 76c038277fc7630a456001dbaf82c77fcb3290fc64a2e2cc6b12374ffb4d3dda
Sep 17 18:08:21.710084 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:21.710055    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n"]
Sep 17 18:08:21.711073 ip-10-0-134-209 kubenswrapper[2581]: W0917 18:08:21.711041    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa9a63c_679c_4161_b889_674a6a4c198e.slice/crio-03b0e0422e455cf25346377b7c4378b305fd8d12b7642e6da0279d2f0edef8e8 WatchSource:0}: Error finding container 03b0e0422e455cf25346377b7c4378b305fd8d12b7642e6da0279d2f0edef8e8: Status 404 returned error can't find the container with id 03b0e0422e455cf25346377b7c4378b305fd8d12b7642e6da0279d2f0edef8e8
Sep 17 18:08:21.946012 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:21.945975    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n" event={"ID":"0fa9a63c-679c-4161-b889-674a6a4c198e","Type":"ContainerStarted","Data":"88fdced58382ccc9d88cd2d50c2222dff3f7185150df4bdb00abbf052614e4b5"}
Sep 17 18:08:21.946521 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:21.946055    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n" event={"ID":"0fa9a63c-679c-4161-b889-674a6a4c198e","Type":"ContainerStarted","Data":"03b0e0422e455cf25346377b7c4378b305fd8d12b7642e6da0279d2f0edef8e8"}
Sep 17 18:08:21.947905 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:21.947878    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn" event={"ID":"2bd93e17-0115-4979-883a-aef312aa68b3","Type":"ContainerStarted","Data":"47e0ab2ae30ee76b1626dfce162d5b33c027f48d48be55df2a2f402bed4a14f0"}
Sep 17 18:08:21.948220 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:21.948193    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv"
Sep 17 18:08:21.948331 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:21.948254    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn" event={"ID":"2bd93e17-0115-4979-883a-aef312aa68b3","Type":"ContainerStarted","Data":"76c038277fc7630a456001dbaf82c77fcb3290fc64a2e2cc6b12374ffb4d3dda"}
Sep 17 18:08:21.949464 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:21.949433    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv" podUID="2a5b7de5-725a-4768-9c72-2e7d04e033c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 18:08:22.952794 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:22.952748    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv" podUID="2a5b7de5-725a-4768-9c72-2e7d04e033c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 18:08:25.967626 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:25.967597    2581 generic.go:356] "Generic (PLEG): container finished" podID="2bd93e17-0115-4979-883a-aef312aa68b3" containerID="47e0ab2ae30ee76b1626dfce162d5b33c027f48d48be55df2a2f402bed4a14f0" exitCode=0
Sep 17 18:08:25.968033 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:25.967680    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn" event={"ID":"2bd93e17-0115-4979-883a-aef312aa68b3","Type":"ContainerDied","Data":"47e0ab2ae30ee76b1626dfce162d5b33c027f48d48be55df2a2f402bed4a14f0"}
Sep 17 18:08:25.969366 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:25.969348    2581 generic.go:356] "Generic (PLEG): container finished" podID="0fa9a63c-679c-4161-b889-674a6a4c198e" containerID="88fdced58382ccc9d88cd2d50c2222dff3f7185150df4bdb00abbf052614e4b5" exitCode=0
Sep 17 18:08:25.969435 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:25.969406    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n" event={"ID":"0fa9a63c-679c-4161-b889-674a6a4c198e","Type":"ContainerDied","Data":"88fdced58382ccc9d88cd2d50c2222dff3f7185150df4bdb00abbf052614e4b5"}
Sep 17 18:08:26.977569 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:26.977529    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn" event={"ID":"2bd93e17-0115-4979-883a-aef312aa68b3","Type":"ContainerStarted","Data":"a1060f1dcb8b294e03c4e99ee4139ebb268721ce3c5aa4e8e4ef7f13562ae655"}
Sep 17 18:08:26.977930 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:26.977613    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn" event={"ID":"2bd93e17-0115-4979-883a-aef312aa68b3","Type":"ContainerStarted","Data":"2d4e311e64b6eff36dcb249bd3786d12409396be4bcc05d5bc751d5b04074f2c"}
Sep 17 18:08:26.977930 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:26.977844    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn"
Sep 17 18:08:26.997306 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:26.997251    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn" podStartSLOduration=3.295204098 podStartE2EDuration="6.997229951s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.702025853s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:26.995962168 +0000 UTC m=+689.096425993" watchObservedRunningTime="2026-09-17 18:08:26.997229951 +0000 UTC m=+689.097693771"
Sep 17 18:08:27.958838 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:27.958812    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv"
Sep 17 18:08:27.959451 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:27.959365    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv" podUID="2a5b7de5-725a-4768-9c72-2e7d04e033c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 18:08:27.985458 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:27.985393    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn"
Sep 17 18:08:27.986935 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:27.986692    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn" podUID="2bd93e17-0115-4979-883a-aef312aa68b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 18:08:28.990038 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:28.989994    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn" podUID="2bd93e17-0115-4979-883a-aef312aa68b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 18:08:33.995733 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:33.995700    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn"
Sep 17 18:08:33.996519 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:33.996456    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn" podUID="2bd93e17-0115-4979-883a-aef312aa68b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 18:08:37.959947 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:37.959911    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv" podUID="2a5b7de5-725a-4768-9c72-2e7d04e033c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 18:08:43.996507 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:43.996423    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn" podUID="2bd93e17-0115-4979-883a-aef312aa68b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 18:08:47.959595 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:47.959556    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv" podUID="2a5b7de5-725a-4768-9c72-2e7d04e033c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 18:08:48.100721 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:48.100682    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n" event={"ID":"0fa9a63c-679c-4161-b889-674a6a4c198e","Type":"ContainerStarted","Data":"98e049e3544827850f26383befd935e6bd7d1cee85d0b51b67eaea4054506d8e"}
Sep 17 18:08:48.100891 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:48.100771    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n" event={"ID":"0fa9a63c-679c-4161-b889-674a6a4c198e","Type":"ContainerStarted","Data":"98fcef7df28fa46c34ed9db94fc5c8464638991a556bbca65d9dacddd611bee4"}
Sep 17 18:08:48.100989 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:48.100970    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n"
Sep 17 18:08:48.122488 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:48.122401    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n" podStartSLOduration=2.795687154 podStartE2EDuration="28.122382481s" totalImagesPullingTime="21.344253937s" totalInitContainerRuntime="3.98244139s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:48.120430551 +0000 UTC m=+710.220894372" watchObservedRunningTime="2026-09-17 18:08:48.122382481 +0000 UTC m=+710.222846302"
Sep 17 18:08:49.105550 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:49.105519    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n"
Sep 17 18:08:49.106448 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:49.106424    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n" podUID="0fa9a63c-679c-4161-b889-674a6a4c198e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 18:08:50.109379 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:50.109345    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n" podUID="0fa9a63c-679c-4161-b889-674a6a4c198e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 18:08:53.996846 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:53.996803    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn" podUID="2bd93e17-0115-4979-883a-aef312aa68b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 18:08:55.113622 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:55.113595    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n"
Sep 17 18:08:55.114094 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:55.114064    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n" podUID="0fa9a63c-679c-4161-b889-674a6a4c198e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 18:08:57.959951 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:08:57.959909    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv" podUID="2a5b7de5-725a-4768-9c72-2e7d04e033c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 18:09:03.997270 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:03.997222    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn" podUID="2bd93e17-0115-4979-883a-aef312aa68b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 18:09:05.114172 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:05.114124    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n" podUID="0fa9a63c-679c-4161-b889-674a6a4c198e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 18:09:07.960206 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:07.960139    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv" podUID="2a5b7de5-725a-4768-9c72-2e7d04e033c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 18:09:13.996512 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:13.996468    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn" podUID="2bd93e17-0115-4979-883a-aef312aa68b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 18:09:15.114568 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:15.114528    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n" podUID="0fa9a63c-679c-4161-b889-674a6a4c198e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 18:09:17.960402 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:17.960365    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv" podUID="2a5b7de5-725a-4768-9c72-2e7d04e033c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 18:09:23.996807 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:23.996765    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn" podUID="2bd93e17-0115-4979-883a-aef312aa68b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 18:09:25.114184 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:25.114127    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n" podUID="0fa9a63c-679c-4161-b889-674a6a4c198e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 18:09:27.960055 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:27.960013    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv" podUID="2a5b7de5-725a-4768-9c72-2e7d04e033c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 18:09:33.996554 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:33.996514    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn" podUID="2bd93e17-0115-4979-883a-aef312aa68b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 18:09:35.114484 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:35.114446    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n" podUID="0fa9a63c-679c-4161-b889-674a6a4c198e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 18:09:37.960594 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:37.960566    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv"
Sep 17 18:09:43.997814 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:43.997783    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn"
Sep 17 18:09:45.114393 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:45.114358    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n" podUID="0fa9a63c-679c-4161-b889-674a6a4c198e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 18:09:45.334482 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:45.334447    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv"]
Sep 17 18:09:45.334721 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:45.334684    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv" podUID="2a5b7de5-725a-4768-9c72-2e7d04e033c0" containerName="kserve-container" containerID="cri-o://b39d7946e30b9d6c37473d412b4a57db3cff1fbce378b2269ea8eaa5202f6fc8" gracePeriod=30
Sep 17 18:09:45.334860 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:45.334728    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv" podUID="2a5b7de5-725a-4768-9c72-2e7d04e033c0" containerName="kube-rbac-proxy" containerID="cri-o://d4752cd229d82fe4cfd22acd8a8b403405724b6633aa1c79d85838725ed26f99" gracePeriod=30
Sep 17 18:09:46.346637 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:46.346600    2581 generic.go:356] "Generic (PLEG): container finished" podID="2a5b7de5-725a-4768-9c72-2e7d04e033c0" containerID="d4752cd229d82fe4cfd22acd8a8b403405724b6633aa1c79d85838725ed26f99" exitCode=2
Sep 17 18:09:46.346983 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:46.346668    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv" event={"ID":"2a5b7de5-725a-4768-9c72-2e7d04e033c0","Type":"ContainerDied","Data":"d4752cd229d82fe4cfd22acd8a8b403405724b6633aa1c79d85838725ed26f99"}
Sep 17 18:09:47.953702 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:47.953659    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv" podUID="2a5b7de5-725a-4768-9c72-2e7d04e033c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 17 18:09:47.959937 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:47.959909    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv" podUID="2a5b7de5-725a-4768-9c72-2e7d04e033c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 18:09:49.769039 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:49.769016    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv"
Sep 17 18:09:49.863792 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:49.863762    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a5b7de5-725a-4768-9c72-2e7d04e033c0-cabundle-cert\") pod \"2a5b7de5-725a-4768-9c72-2e7d04e033c0\" (UID: \"2a5b7de5-725a-4768-9c72-2e7d04e033c0\") "
Sep 17 18:09:49.863941 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:49.863796    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfs5q\" (UniqueName: \"kubernetes.io/projected/2a5b7de5-725a-4768-9c72-2e7d04e033c0-kube-api-access-xfs5q\") pod \"2a5b7de5-725a-4768-9c72-2e7d04e033c0\" (UID: \"2a5b7de5-725a-4768-9c72-2e7d04e033c0\") "
Sep 17 18:09:49.863941 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:49.863820    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a5b7de5-725a-4768-9c72-2e7d04e033c0-kube-rbac-proxy-sar-config\") pod \"2a5b7de5-725a-4768-9c72-2e7d04e033c0\" (UID: \"2a5b7de5-725a-4768-9c72-2e7d04e033c0\") "
Sep 17 18:09:49.863941 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:49.863845    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a5b7de5-725a-4768-9c72-2e7d04e033c0-kserve-provision-location\") pod \"2a5b7de5-725a-4768-9c72-2e7d04e033c0\" (UID: \"2a5b7de5-725a-4768-9c72-2e7d04e033c0\") "
Sep 17 18:09:49.863941 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:49.863903    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a5b7de5-725a-4768-9c72-2e7d04e033c0-proxy-tls\") pod \"2a5b7de5-725a-4768-9c72-2e7d04e033c0\" (UID: \"2a5b7de5-725a-4768-9c72-2e7d04e033c0\") "
Sep 17 18:09:49.864209 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:49.864182    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a5b7de5-725a-4768-9c72-2e7d04e033c0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2a5b7de5-725a-4768-9c72-2e7d04e033c0" (UID: "2a5b7de5-725a-4768-9c72-2e7d04e033c0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:09:49.864209 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:49.864197    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a5b7de5-725a-4768-9c72-2e7d04e033c0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2a5b7de5-725a-4768-9c72-2e7d04e033c0" (UID: "2a5b7de5-725a-4768-9c72-2e7d04e033c0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:09:49.864302 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:49.864235    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a5b7de5-725a-4768-9c72-2e7d04e033c0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2a5b7de5-725a-4768-9c72-2e7d04e033c0" (UID: "2a5b7de5-725a-4768-9c72-2e7d04e033c0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:09:49.865966 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:49.865943    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a5b7de5-725a-4768-9c72-2e7d04e033c0-kube-api-access-xfs5q" (OuterVolumeSpecName: "kube-api-access-xfs5q") pod "2a5b7de5-725a-4768-9c72-2e7d04e033c0" (UID: "2a5b7de5-725a-4768-9c72-2e7d04e033c0"). InnerVolumeSpecName "kube-api-access-xfs5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:09:49.866058 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:49.866010    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a5b7de5-725a-4768-9c72-2e7d04e033c0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2a5b7de5-725a-4768-9c72-2e7d04e033c0" (UID: "2a5b7de5-725a-4768-9c72-2e7d04e033c0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:09:49.965038 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:49.964972    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a5b7de5-725a-4768-9c72-2e7d04e033c0-proxy-tls\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:09:49.965038 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:49.964995    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a5b7de5-725a-4768-9c72-2e7d04e033c0-cabundle-cert\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:09:49.965038 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:49.965008    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfs5q\" (UniqueName: \"kubernetes.io/projected/2a5b7de5-725a-4768-9c72-2e7d04e033c0-kube-api-access-xfs5q\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:09:49.965038 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:49.965017    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a5b7de5-725a-4768-9c72-2e7d04e033c0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:09:49.965038 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:49.965025    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a5b7de5-725a-4768-9c72-2e7d04e033c0-kserve-provision-location\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:09:50.368456 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.368428    2581 generic.go:356] "Generic (PLEG): container finished" podID="2a5b7de5-725a-4768-9c72-2e7d04e033c0" containerID="b39d7946e30b9d6c37473d412b4a57db3cff1fbce378b2269ea8eaa5202f6fc8" exitCode=0
Sep 17 18:09:50.368605 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.368491    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv"
Sep 17 18:09:50.368605 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.368503    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv" event={"ID":"2a5b7de5-725a-4768-9c72-2e7d04e033c0","Type":"ContainerDied","Data":"b39d7946e30b9d6c37473d412b4a57db3cff1fbce378b2269ea8eaa5202f6fc8"}
Sep 17 18:09:50.368605 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.368566    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv" event={"ID":"2a5b7de5-725a-4768-9c72-2e7d04e033c0","Type":"ContainerDied","Data":"d654b1cf8749bdf1452fdbe89100d063a76f3493bc85ae8343242e0135c4a5ed"}
Sep 17 18:09:50.368605 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.368594    2581 scope.go:117] "RemoveContainer" containerID="d4752cd229d82fe4cfd22acd8a8b403405724b6633aa1c79d85838725ed26f99"
Sep 17 18:09:50.376764 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.376748    2581 scope.go:117] "RemoveContainer" containerID="b39d7946e30b9d6c37473d412b4a57db3cff1fbce378b2269ea8eaa5202f6fc8"
Sep 17 18:09:50.384360 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.384345    2581 scope.go:117] "RemoveContainer" containerID="7ba513683e56e7246a7e78577c124b7561f7f9377d4960545ce9253b6b21126b"
Sep 17 18:09:50.390141 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.390121    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv"]
Sep 17 18:09:50.392452 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.392434    2581 scope.go:117] "RemoveContainer" containerID="d4752cd229d82fe4cfd22acd8a8b403405724b6633aa1c79d85838725ed26f99"
Sep 17 18:09:50.392710 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:09:50.392689    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4752cd229d82fe4cfd22acd8a8b403405724b6633aa1c79d85838725ed26f99\": container with ID starting with d4752cd229d82fe4cfd22acd8a8b403405724b6633aa1c79d85838725ed26f99 not found: ID does not exist" containerID="d4752cd229d82fe4cfd22acd8a8b403405724b6633aa1c79d85838725ed26f99"
Sep 17 18:09:50.392778 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.392722    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4752cd229d82fe4cfd22acd8a8b403405724b6633aa1c79d85838725ed26f99"} err="failed to get container status \"d4752cd229d82fe4cfd22acd8a8b403405724b6633aa1c79d85838725ed26f99\": rpc error: code = NotFound desc = could not find container \"d4752cd229d82fe4cfd22acd8a8b403405724b6633aa1c79d85838725ed26f99\": container with ID starting with d4752cd229d82fe4cfd22acd8a8b403405724b6633aa1c79d85838725ed26f99 not found: ID does not exist"
Sep 17 18:09:50.392778 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.392744    2581 scope.go:117] "RemoveContainer" containerID="b39d7946e30b9d6c37473d412b4a57db3cff1fbce378b2269ea8eaa5202f6fc8"
Sep 17 18:09:50.393027 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:09:50.393000    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b39d7946e30b9d6c37473d412b4a57db3cff1fbce378b2269ea8eaa5202f6fc8\": container with ID starting with b39d7946e30b9d6c37473d412b4a57db3cff1fbce378b2269ea8eaa5202f6fc8 not found: ID does not exist" containerID="b39d7946e30b9d6c37473d412b4a57db3cff1fbce378b2269ea8eaa5202f6fc8"
Sep 17 18:09:50.393125 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.393029    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b39d7946e30b9d6c37473d412b4a57db3cff1fbce378b2269ea8eaa5202f6fc8"} err="failed to get container status \"b39d7946e30b9d6c37473d412b4a57db3cff1fbce378b2269ea8eaa5202f6fc8\": rpc error: code = NotFound desc = could not find container \"b39d7946e30b9d6c37473d412b4a57db3cff1fbce378b2269ea8eaa5202f6fc8\": container with ID starting with b39d7946e30b9d6c37473d412b4a57db3cff1fbce378b2269ea8eaa5202f6fc8 not found: ID does not exist"
Sep 17 18:09:50.393125 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.393056    2581 scope.go:117] "RemoveContainer" containerID="7ba513683e56e7246a7e78577c124b7561f7f9377d4960545ce9253b6b21126b"
Sep 17 18:09:50.393374 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:09:50.393352    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba513683e56e7246a7e78577c124b7561f7f9377d4960545ce9253b6b21126b\": container with ID starting with 7ba513683e56e7246a7e78577c124b7561f7f9377d4960545ce9253b6b21126b not found: ID does not exist" containerID="7ba513683e56e7246a7e78577c124b7561f7f9377d4960545ce9253b6b21126b"
Sep 17 18:09:50.393453 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.393384    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba513683e56e7246a7e78577c124b7561f7f9377d4960545ce9253b6b21126b"} err="failed to get container status \"7ba513683e56e7246a7e78577c124b7561f7f9377d4960545ce9253b6b21126b\": rpc error: code = NotFound desc = could not find container \"7ba513683e56e7246a7e78577c124b7561f7f9377d4960545ce9253b6b21126b\": container with ID starting with 7ba513683e56e7246a7e78577c124b7561f7f9377d4960545ce9253b6b21126b not found: ID does not exist"
Sep 17 18:09:50.395313 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.395294    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-nrzjv"]
Sep 17 18:09:50.544545 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.544510    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8"]
Sep 17 18:09:50.544998 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.544984    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a5b7de5-725a-4768-9c72-2e7d04e033c0" containerName="kserve-container"
Sep 17 18:09:50.545039 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.545002    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a5b7de5-725a-4768-9c72-2e7d04e033c0" containerName="kserve-container"
Sep 17 18:09:50.545039 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.545025    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a5b7de5-725a-4768-9c72-2e7d04e033c0" containerName="storage-initializer"
Sep 17 18:09:50.545039 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.545034    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a5b7de5-725a-4768-9c72-2e7d04e033c0" containerName="storage-initializer"
Sep 17 18:09:50.545128 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.545054    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a5b7de5-725a-4768-9c72-2e7d04e033c0" containerName="kube-rbac-proxy"
Sep 17 18:09:50.545128 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.545063    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a5b7de5-725a-4768-9c72-2e7d04e033c0" containerName="kube-rbac-proxy"
Sep 17 18:09:50.545210 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.545134    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a5b7de5-725a-4768-9c72-2e7d04e033c0" containerName="kube-rbac-proxy"
Sep 17 18:09:50.545210 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.545164    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a5b7de5-725a-4768-9c72-2e7d04e033c0" containerName="kserve-container"
Sep 17 18:09:50.548344 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.548325    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8"
Sep 17 18:09:50.550292 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.550268    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 18:09:50.550438 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.550343    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 18:09:50.559322 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.559294    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8"]
Sep 17 18:09:50.568858 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.568836    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4edabf76-33a1-495b-a1a7-c5c36c0b3929-cabundle-cert\") pod \"isvc-canary-stop-predictor-b84cdfc7-c8kb8\" (UID: \"4edabf76-33a1-495b-a1a7-c5c36c0b3929\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8"
Sep 17 18:09:50.568962 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.568875    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4edabf76-33a1-495b-a1a7-c5c36c0b3929-kserve-provision-location\") pod \"isvc-canary-stop-predictor-b84cdfc7-c8kb8\" (UID: \"4edabf76-33a1-495b-a1a7-c5c36c0b3929\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8"
Sep 17 18:09:50.569019 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.568967    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4edabf76-33a1-495b-a1a7-c5c36c0b3929-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-b84cdfc7-c8kb8\" (UID: \"4edabf76-33a1-495b-a1a7-c5c36c0b3929\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8"
Sep 17 18:09:50.569019 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.568996    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4edabf76-33a1-495b-a1a7-c5c36c0b3929-proxy-tls\") pod \"isvc-canary-stop-predictor-b84cdfc7-c8kb8\" (UID: \"4edabf76-33a1-495b-a1a7-c5c36c0b3929\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8"
Sep 17 18:09:50.569128 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.569058    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn8f2\" (UniqueName: \"kubernetes.io/projected/4edabf76-33a1-495b-a1a7-c5c36c0b3929-kube-api-access-pn8f2\") pod \"isvc-canary-stop-predictor-b84cdfc7-c8kb8\" (UID: \"4edabf76-33a1-495b-a1a7-c5c36c0b3929\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8"
Sep 17 18:09:50.669780 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.669703    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4edabf76-33a1-495b-a1a7-c5c36c0b3929-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-b84cdfc7-c8kb8\" (UID: \"4edabf76-33a1-495b-a1a7-c5c36c0b3929\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8"
Sep 17 18:09:50.669780 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.669742    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4edabf76-33a1-495b-a1a7-c5c36c0b3929-proxy-tls\") pod \"isvc-canary-stop-predictor-b84cdfc7-c8kb8\" (UID: \"4edabf76-33a1-495b-a1a7-c5c36c0b3929\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8"
Sep 17 18:09:50.669974 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:09:50.669861    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 18:09:50.669974 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.669876    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn8f2\" (UniqueName: \"kubernetes.io/projected/4edabf76-33a1-495b-a1a7-c5c36c0b3929-kube-api-access-pn8f2\") pod \"isvc-canary-stop-predictor-b84cdfc7-c8kb8\" (UID: \"4edabf76-33a1-495b-a1a7-c5c36c0b3929\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8"
Sep 17 18:09:50.669974 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:09:50.669917    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4edabf76-33a1-495b-a1a7-c5c36c0b3929-proxy-tls podName:4edabf76-33a1-495b-a1a7-c5c36c0b3929 nodeName:}" failed. No retries permitted until 2026-09-17 18:09:51.169898888 +0000 UTC m=+773.270362691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4edabf76-33a1-495b-a1a7-c5c36c0b3929-proxy-tls") pod "isvc-canary-stop-predictor-b84cdfc7-c8kb8" (UID: "4edabf76-33a1-495b-a1a7-c5c36c0b3929") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 18:09:50.670172 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.669981    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4edabf76-33a1-495b-a1a7-c5c36c0b3929-cabundle-cert\") pod \"isvc-canary-stop-predictor-b84cdfc7-c8kb8\" (UID: \"4edabf76-33a1-495b-a1a7-c5c36c0b3929\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8"
Sep 17 18:09:50.670172 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.670017    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4edabf76-33a1-495b-a1a7-c5c36c0b3929-kserve-provision-location\") pod \"isvc-canary-stop-predictor-b84cdfc7-c8kb8\" (UID: \"4edabf76-33a1-495b-a1a7-c5c36c0b3929\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8"
Sep 17 18:09:50.670389 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.670369    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4edabf76-33a1-495b-a1a7-c5c36c0b3929-kserve-provision-location\") pod \"isvc-canary-stop-predictor-b84cdfc7-c8kb8\" (UID: \"4edabf76-33a1-495b-a1a7-c5c36c0b3929\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8"
Sep 17 18:09:50.670533 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.670511    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4edabf76-33a1-495b-a1a7-c5c36c0b3929-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-b84cdfc7-c8kb8\" (UID: \"4edabf76-33a1-495b-a1a7-c5c36c0b3929\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8"
Sep 17 18:09:50.670579 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.670551    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4edabf76-33a1-495b-a1a7-c5c36c0b3929-cabundle-cert\") pod \"isvc-canary-stop-predictor-b84cdfc7-c8kb8\" (UID: \"4edabf76-33a1-495b-a1a7-c5c36c0b3929\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8"
Sep 17 18:09:50.677967 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:50.677941    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn8f2\" (UniqueName: \"kubernetes.io/projected/4edabf76-33a1-495b-a1a7-c5c36c0b3929-kube-api-access-pn8f2\") pod \"isvc-canary-stop-predictor-b84cdfc7-c8kb8\" (UID: \"4edabf76-33a1-495b-a1a7-c5c36c0b3929\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8"
Sep 17 18:09:51.175447 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:51.175416    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4edabf76-33a1-495b-a1a7-c5c36c0b3929-proxy-tls\") pod \"isvc-canary-stop-predictor-b84cdfc7-c8kb8\" (UID: \"4edabf76-33a1-495b-a1a7-c5c36c0b3929\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8"
Sep 17 18:09:51.177890 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:51.177861    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4edabf76-33a1-495b-a1a7-c5c36c0b3929-proxy-tls\") pod \"isvc-canary-stop-predictor-b84cdfc7-c8kb8\" (UID: \"4edabf76-33a1-495b-a1a7-c5c36c0b3929\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8"
Sep 17 18:09:51.460309 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:51.460238    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8"
Sep 17 18:09:51.736668 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:51.736599    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8"]
Sep 17 18:09:51.750789 ip-10-0-134-209 kubenswrapper[2581]: W0917 18:09:51.750752    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4edabf76_33a1_495b_a1a7_c5c36c0b3929.slice/crio-b0d60783b655a78d98be2eff11fa246c0f23cc1ba59741013f3bcbb2b984574e WatchSource:0}: Error finding container b0d60783b655a78d98be2eff11fa246c0f23cc1ba59741013f3bcbb2b984574e: Status 404 returned error can't find the container with id b0d60783b655a78d98be2eff11fa246c0f23cc1ba59741013f3bcbb2b984574e
Sep 17 18:09:52.049332 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:52.049232    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a5b7de5-725a-4768-9c72-2e7d04e033c0" path="/var/lib/kubelet/pods/2a5b7de5-725a-4768-9c72-2e7d04e033c0/volumes"
Sep 17 18:09:52.379210 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:52.379171    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8" event={"ID":"4edabf76-33a1-495b-a1a7-c5c36c0b3929","Type":"ContainerStarted","Data":"731e62e4726b62f6b46f569a94704615715c00c98db637a4488a4d56e55c98e5"}
Sep 17 18:09:52.379561 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:52.379239    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8" event={"ID":"4edabf76-33a1-495b-a1a7-c5c36c0b3929","Type":"ContainerStarted","Data":"b0d60783b655a78d98be2eff11fa246c0f23cc1ba59741013f3bcbb2b984574e"}
Sep 17 18:09:55.114308 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:55.114280    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n"
Sep 17 18:09:56.397378 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:56.397345    2581 generic.go:356] "Generic (PLEG): container finished" podID="4edabf76-33a1-495b-a1a7-c5c36c0b3929" containerID="731e62e4726b62f6b46f569a94704615715c00c98db637a4488a4d56e55c98e5" exitCode=0
Sep 17 18:09:56.397810 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:56.397430    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8" event={"ID":"4edabf76-33a1-495b-a1a7-c5c36c0b3929","Type":"ContainerDied","Data":"731e62e4726b62f6b46f569a94704615715c00c98db637a4488a4d56e55c98e5"}
Sep 17 18:09:57.403976 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:57.403940    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8" event={"ID":"4edabf76-33a1-495b-a1a7-c5c36c0b3929","Type":"ContainerStarted","Data":"eb832e09058afb1eaad800aa60cc90118111d8904a122af43b4c2cdf6b30e6f0"}
Sep 17 18:09:57.404364 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:57.404028    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8" event={"ID":"4edabf76-33a1-495b-a1a7-c5c36c0b3929","Type":"ContainerStarted","Data":"284df10069fc76e87b52c91fde0f3c5a0dd6f9115fb82cd196277deec0e0bf9d"}
Sep 17 18:09:57.404364 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:57.404294    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8"
Sep 17 18:09:57.433095 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:57.433042    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8" podStartSLOduration=3.4895430210000002 podStartE2EDuration="7.433023397s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.943480376s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:09:57.431542312 +0000 UTC m=+779.532006133" watchObservedRunningTime="2026-09-17 18:09:57.433023397 +0000 UTC m=+779.533487217"
Sep 17 18:09:58.408754 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:58.408719    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8"
Sep 17 18:09:58.410038 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:58.410012    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8" podUID="4edabf76-33a1-495b-a1a7-c5c36c0b3929" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 18:09:59.414145 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:09:59.414105    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8" podUID="4edabf76-33a1-495b-a1a7-c5c36c0b3929" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 18:10:04.418817 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:04.418790    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8"
Sep 17 18:10:04.419336 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:04.419312    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8" podUID="4edabf76-33a1-495b-a1a7-c5c36c0b3929" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 18:10:10.594019 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:10.593935    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn"]
Sep 17 18:10:10.594469 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:10.594262    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn" podUID="2bd93e17-0115-4979-883a-aef312aa68b3" containerName="kserve-container" containerID="cri-o://2d4e311e64b6eff36dcb249bd3786d12409396be4bcc05d5bc751d5b04074f2c" gracePeriod=30
Sep 17 18:10:10.594469 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:10.594327    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn" podUID="2bd93e17-0115-4979-883a-aef312aa68b3" containerName="kube-rbac-proxy" containerID="cri-o://a1060f1dcb8b294e03c4e99ee4139ebb268721ce3c5aa4e8e4ef7f13562ae655" gracePeriod=30
Sep 17 18:10:10.634484 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:10.634460    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n"]
Sep 17 18:10:10.634699 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:10.634677    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n" podUID="0fa9a63c-679c-4161-b889-674a6a4c198e" containerName="kserve-container" containerID="cri-o://98fcef7df28fa46c34ed9db94fc5c8464638991a556bbca65d9dacddd611bee4" gracePeriod=30
Sep 17 18:10:10.634782 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:10.634732    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n" podUID="0fa9a63c-679c-4161-b889-674a6a4c198e" containerName="kube-rbac-proxy" containerID="cri-o://98e049e3544827850f26383befd935e6bd7d1cee85d0b51b67eaea4054506d8e" gracePeriod=30
Sep 17 18:10:11.467536 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:11.467504    2581 generic.go:356] "Generic (PLEG): container finished" podID="0fa9a63c-679c-4161-b889-674a6a4c198e" containerID="98e049e3544827850f26383befd935e6bd7d1cee85d0b51b67eaea4054506d8e" exitCode=2
Sep 17 18:10:11.467723 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:11.467579    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n" event={"ID":"0fa9a63c-679c-4161-b889-674a6a4c198e","Type":"ContainerDied","Data":"98e049e3544827850f26383befd935e6bd7d1cee85d0b51b67eaea4054506d8e"}
Sep 17 18:10:11.469966 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:11.469946    2581 generic.go:356] "Generic (PLEG): container finished" podID="2bd93e17-0115-4979-883a-aef312aa68b3" containerID="a1060f1dcb8b294e03c4e99ee4139ebb268721ce3c5aa4e8e4ef7f13562ae655" exitCode=2
Sep 17 18:10:11.470076 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:11.470016    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn" event={"ID":"2bd93e17-0115-4979-883a-aef312aa68b3","Type":"ContainerDied","Data":"a1060f1dcb8b294e03c4e99ee4139ebb268721ce3c5aa4e8e4ef7f13562ae655"}
Sep 17 18:10:13.991238 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:13.991199    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn" podUID="2bd93e17-0115-4979-883a-aef312aa68b3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 17 18:10:13.997128 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:13.997103    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn" podUID="2bd93e17-0115-4979-883a-aef312aa68b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 18:10:14.419256 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:14.419218    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8" podUID="4edabf76-33a1-495b-a1a7-c5c36c0b3929" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 18:10:14.485372 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:14.485344    2581 generic.go:356] "Generic (PLEG): container finished" podID="0fa9a63c-679c-4161-b889-674a6a4c198e" containerID="98fcef7df28fa46c34ed9db94fc5c8464638991a556bbca65d9dacddd611bee4" exitCode=0
Sep 17 18:10:14.485507 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:14.485414    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n" event={"ID":"0fa9a63c-679c-4161-b889-674a6a4c198e","Type":"ContainerDied","Data":"98fcef7df28fa46c34ed9db94fc5c8464638991a556bbca65d9dacddd611bee4"}
Sep 17 18:10:14.577803 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:14.577775    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n"
Sep 17 18:10:14.675485 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:14.675416    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0fa9a63c-679c-4161-b889-674a6a4c198e-kube-rbac-proxy-sar-config\") pod \"0fa9a63c-679c-4161-b889-674a6a4c198e\" (UID: \"0fa9a63c-679c-4161-b889-674a6a4c198e\") "
Sep 17 18:10:14.675485 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:14.675452    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmtrs\" (UniqueName: \"kubernetes.io/projected/0fa9a63c-679c-4161-b889-674a6a4c198e-kube-api-access-nmtrs\") pod \"0fa9a63c-679c-4161-b889-674a6a4c198e\" (UID: \"0fa9a63c-679c-4161-b889-674a6a4c198e\") "
Sep 17 18:10:14.675485 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:14.675478    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0fa9a63c-679c-4161-b889-674a6a4c198e-cabundle-cert\") pod \"0fa9a63c-679c-4161-b889-674a6a4c198e\" (UID: \"0fa9a63c-679c-4161-b889-674a6a4c198e\") "
Sep 17 18:10:14.675728 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:14.675583    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fa9a63c-679c-4161-b889-674a6a4c198e-kserve-provision-location\") pod \"0fa9a63c-679c-4161-b889-674a6a4c198e\" (UID: \"0fa9a63c-679c-4161-b889-674a6a4c198e\") "
Sep 17 18:10:14.675728 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:14.675657    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fa9a63c-679c-4161-b889-674a6a4c198e-proxy-tls\") pod \"0fa9a63c-679c-4161-b889-674a6a4c198e\" (UID: \"0fa9a63c-679c-4161-b889-674a6a4c198e\") "
Sep 17 18:10:14.675927 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:14.675897    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa9a63c-679c-4161-b889-674a6a4c198e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0fa9a63c-679c-4161-b889-674a6a4c198e" (UID: "0fa9a63c-679c-4161-b889-674a6a4c198e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:10:14.675927 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:14.675912    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa9a63c-679c-4161-b889-674a6a4c198e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0fa9a63c-679c-4161-b889-674a6a4c198e" (UID: "0fa9a63c-679c-4161-b889-674a6a4c198e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:10:14.676054 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:14.675926    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fa9a63c-679c-4161-b889-674a6a4c198e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0fa9a63c-679c-4161-b889-674a6a4c198e" (UID: "0fa9a63c-679c-4161-b889-674a6a4c198e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:14.677748 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:14.677721    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fa9a63c-679c-4161-b889-674a6a4c198e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0fa9a63c-679c-4161-b889-674a6a4c198e" (UID: "0fa9a63c-679c-4161-b889-674a6a4c198e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:10:14.677817 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:14.677798    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fa9a63c-679c-4161-b889-674a6a4c198e-kube-api-access-nmtrs" (OuterVolumeSpecName: "kube-api-access-nmtrs") pod "0fa9a63c-679c-4161-b889-674a6a4c198e" (UID: "0fa9a63c-679c-4161-b889-674a6a4c198e"). InnerVolumeSpecName "kube-api-access-nmtrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:10:14.776919 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:14.776883    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fa9a63c-679c-4161-b889-674a6a4c198e-proxy-tls\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:10:14.776919 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:14.776912    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0fa9a63c-679c-4161-b889-674a6a4c198e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:10:14.776919 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:14.776923    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmtrs\" (UniqueName: \"kubernetes.io/projected/0fa9a63c-679c-4161-b889-674a6a4c198e-kube-api-access-nmtrs\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:10:14.777229 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:14.776932    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0fa9a63c-679c-4161-b889-674a6a4c198e-cabundle-cert\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:10:14.777229 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:14.776942    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fa9a63c-679c-4161-b889-674a6a4c198e-kserve-provision-location\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:10:15.029049 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.029025    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn"
Sep 17 18:10:15.180540 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.180512    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bd93e17-0115-4979-883a-aef312aa68b3-cabundle-cert\") pod \"2bd93e17-0115-4979-883a-aef312aa68b3\" (UID: \"2bd93e17-0115-4979-883a-aef312aa68b3\") "
Sep 17 18:10:15.180723 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.180560    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8vfl\" (UniqueName: \"kubernetes.io/projected/2bd93e17-0115-4979-883a-aef312aa68b3-kube-api-access-l8vfl\") pod \"2bd93e17-0115-4979-883a-aef312aa68b3\" (UID: \"2bd93e17-0115-4979-883a-aef312aa68b3\") "
Sep 17 18:10:15.180723 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.180580    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bd93e17-0115-4979-883a-aef312aa68b3-kserve-provision-location\") pod \"2bd93e17-0115-4979-883a-aef312aa68b3\" (UID: \"2bd93e17-0115-4979-883a-aef312aa68b3\") "
Sep 17 18:10:15.180723 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.180597    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bd93e17-0115-4979-883a-aef312aa68b3-kube-rbac-proxy-sar-config\") pod \"2bd93e17-0115-4979-883a-aef312aa68b3\" (UID: \"2bd93e17-0115-4979-883a-aef312aa68b3\") "
Sep 17 18:10:15.180878 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.180719    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bd93e17-0115-4979-883a-aef312aa68b3-proxy-tls\") pod \"2bd93e17-0115-4979-883a-aef312aa68b3\" (UID: \"2bd93e17-0115-4979-883a-aef312aa68b3\") "
Sep 17 18:10:15.180929 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.180876    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd93e17-0115-4979-883a-aef312aa68b3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2bd93e17-0115-4979-883a-aef312aa68b3" (UID: "2bd93e17-0115-4979-883a-aef312aa68b3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:15.180929 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.180908    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd93e17-0115-4979-883a-aef312aa68b3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2bd93e17-0115-4979-883a-aef312aa68b3" (UID: "2bd93e17-0115-4979-883a-aef312aa68b3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:10:15.180929 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.180914    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd93e17-0115-4979-883a-aef312aa68b3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2bd93e17-0115-4979-883a-aef312aa68b3" (UID: "2bd93e17-0115-4979-883a-aef312aa68b3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:10:15.181131 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.181110    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bd93e17-0115-4979-883a-aef312aa68b3-kserve-provision-location\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:10:15.181222 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.181137    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bd93e17-0115-4979-883a-aef312aa68b3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:10:15.181222 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.181177    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bd93e17-0115-4979-883a-aef312aa68b3-cabundle-cert\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:10:15.182775 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.182753    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd93e17-0115-4979-883a-aef312aa68b3-kube-api-access-l8vfl" (OuterVolumeSpecName: "kube-api-access-l8vfl") pod "2bd93e17-0115-4979-883a-aef312aa68b3" (UID: "2bd93e17-0115-4979-883a-aef312aa68b3"). InnerVolumeSpecName "kube-api-access-l8vfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:10:15.182892 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.182874    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd93e17-0115-4979-883a-aef312aa68b3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2bd93e17-0115-4979-883a-aef312aa68b3" (UID: "2bd93e17-0115-4979-883a-aef312aa68b3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:10:15.282540 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.282443    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8vfl\" (UniqueName: \"kubernetes.io/projected/2bd93e17-0115-4979-883a-aef312aa68b3-kube-api-access-l8vfl\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:10:15.282540 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.282486    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bd93e17-0115-4979-883a-aef312aa68b3-proxy-tls\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:10:15.491462 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.491439    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n"
Sep 17 18:10:15.491632 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.491472    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n" event={"ID":"0fa9a63c-679c-4161-b889-674a6a4c198e","Type":"ContainerDied","Data":"03b0e0422e455cf25346377b7c4378b305fd8d12b7642e6da0279d2f0edef8e8"}
Sep 17 18:10:15.491632 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.491553    2581 scope.go:117] "RemoveContainer" containerID="98e049e3544827850f26383befd935e6bd7d1cee85d0b51b67eaea4054506d8e"
Sep 17 18:10:15.493454 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.493428    2581 generic.go:356] "Generic (PLEG): container finished" podID="2bd93e17-0115-4979-883a-aef312aa68b3" containerID="2d4e311e64b6eff36dcb249bd3786d12409396be4bcc05d5bc751d5b04074f2c" exitCode=0
Sep 17 18:10:15.493566 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.493485    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn"
Sep 17 18:10:15.493566 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.493521    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn" event={"ID":"2bd93e17-0115-4979-883a-aef312aa68b3","Type":"ContainerDied","Data":"2d4e311e64b6eff36dcb249bd3786d12409396be4bcc05d5bc751d5b04074f2c"}
Sep 17 18:10:15.493682 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.493586    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn" event={"ID":"2bd93e17-0115-4979-883a-aef312aa68b3","Type":"ContainerDied","Data":"76c038277fc7630a456001dbaf82c77fcb3290fc64a2e2cc6b12374ffb4d3dda"}
Sep 17 18:10:15.500864 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.500832    2581 scope.go:117] "RemoveContainer" containerID="98fcef7df28fa46c34ed9db94fc5c8464638991a556bbca65d9dacddd611bee4"
Sep 17 18:10:15.508476 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.508460    2581 scope.go:117] "RemoveContainer" containerID="88fdced58382ccc9d88cd2d50c2222dff3f7185150df4bdb00abbf052614e4b5"
Sep 17 18:10:15.514931 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.514909    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n"]
Sep 17 18:10:15.515974 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.515958    2581 scope.go:117] "RemoveContainer" containerID="a1060f1dcb8b294e03c4e99ee4139ebb268721ce3c5aa4e8e4ef7f13562ae655"
Sep 17 18:10:15.518385 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.518366    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-0cba7-predictor-569fd45685-slt9n"]
Sep 17 18:10:15.524900 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.524884    2581 scope.go:117] "RemoveContainer" containerID="2d4e311e64b6eff36dcb249bd3786d12409396be4bcc05d5bc751d5b04074f2c"
Sep 17 18:10:15.529304 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.529283    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn"]
Sep 17 18:10:15.533007 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.532984    2581 scope.go:117] "RemoveContainer" containerID="47e0ab2ae30ee76b1626dfce162d5b33c027f48d48be55df2a2f402bed4a14f0"
Sep 17 18:10:15.533671 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.533654    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-0cba7-predictor-8cbb5dcb4-znrvn"]
Sep 17 18:10:15.540481 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.540463    2581 scope.go:117] "RemoveContainer" containerID="a1060f1dcb8b294e03c4e99ee4139ebb268721ce3c5aa4e8e4ef7f13562ae655"
Sep 17 18:10:15.540727 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:10:15.540708    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1060f1dcb8b294e03c4e99ee4139ebb268721ce3c5aa4e8e4ef7f13562ae655\": container with ID starting with a1060f1dcb8b294e03c4e99ee4139ebb268721ce3c5aa4e8e4ef7f13562ae655 not found: ID does not exist" containerID="a1060f1dcb8b294e03c4e99ee4139ebb268721ce3c5aa4e8e4ef7f13562ae655"
Sep 17 18:10:15.540772 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.540737    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1060f1dcb8b294e03c4e99ee4139ebb268721ce3c5aa4e8e4ef7f13562ae655"} err="failed to get container status \"a1060f1dcb8b294e03c4e99ee4139ebb268721ce3c5aa4e8e4ef7f13562ae655\": rpc error: code = NotFound desc = could not find container \"a1060f1dcb8b294e03c4e99ee4139ebb268721ce3c5aa4e8e4ef7f13562ae655\": container with ID starting with a1060f1dcb8b294e03c4e99ee4139ebb268721ce3c5aa4e8e4ef7f13562ae655 not found: ID does not exist"
Sep 17 18:10:15.540772 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.540755    2581 scope.go:117] "RemoveContainer" containerID="2d4e311e64b6eff36dcb249bd3786d12409396be4bcc05d5bc751d5b04074f2c"
Sep 17 18:10:15.541007 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:10:15.540992    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d4e311e64b6eff36dcb249bd3786d12409396be4bcc05d5bc751d5b04074f2c\": container with ID starting with 2d4e311e64b6eff36dcb249bd3786d12409396be4bcc05d5bc751d5b04074f2c not found: ID does not exist" containerID="2d4e311e64b6eff36dcb249bd3786d12409396be4bcc05d5bc751d5b04074f2c"
Sep 17 18:10:15.541062 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.541010    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d4e311e64b6eff36dcb249bd3786d12409396be4bcc05d5bc751d5b04074f2c"} err="failed to get container status \"2d4e311e64b6eff36dcb249bd3786d12409396be4bcc05d5bc751d5b04074f2c\": rpc error: code = NotFound desc = could not find container \"2d4e311e64b6eff36dcb249bd3786d12409396be4bcc05d5bc751d5b04074f2c\": container with ID starting with 2d4e311e64b6eff36dcb249bd3786d12409396be4bcc05d5bc751d5b04074f2c not found: ID does not exist"
Sep 17 18:10:15.541062 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.541022    2581 scope.go:117] "RemoveContainer" containerID="47e0ab2ae30ee76b1626dfce162d5b33c027f48d48be55df2a2f402bed4a14f0"
Sep 17 18:10:15.541238 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:10:15.541223    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47e0ab2ae30ee76b1626dfce162d5b33c027f48d48be55df2a2f402bed4a14f0\": container with ID starting with 47e0ab2ae30ee76b1626dfce162d5b33c027f48d48be55df2a2f402bed4a14f0 not found: ID does not exist" containerID="47e0ab2ae30ee76b1626dfce162d5b33c027f48d48be55df2a2f402bed4a14f0"
Sep 17 18:10:15.541290 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:15.541242    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47e0ab2ae30ee76b1626dfce162d5b33c027f48d48be55df2a2f402bed4a14f0"} err="failed to get container status \"47e0ab2ae30ee76b1626dfce162d5b33c027f48d48be55df2a2f402bed4a14f0\": rpc error: code = NotFound desc = could not find container \"47e0ab2ae30ee76b1626dfce162d5b33c027f48d48be55df2a2f402bed4a14f0\": container with ID starting with 47e0ab2ae30ee76b1626dfce162d5b33c027f48d48be55df2a2f402bed4a14f0 not found: ID does not exist"
Sep 17 18:10:16.049019 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:16.048988    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fa9a63c-679c-4161-b889-674a6a4c198e" path="/var/lib/kubelet/pods/0fa9a63c-679c-4161-b889-674a6a4c198e/volumes"
Sep 17 18:10:16.049502 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:16.049488    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bd93e17-0115-4979-883a-aef312aa68b3" path="/var/lib/kubelet/pods/2bd93e17-0115-4979-883a-aef312aa68b3/volumes"
Sep 17 18:10:24.419470 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:24.419424    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8" podUID="4edabf76-33a1-495b-a1a7-c5c36c0b3929" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 18:10:34.419895 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:34.419857    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8" podUID="4edabf76-33a1-495b-a1a7-c5c36c0b3929" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 18:10:44.419851 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:44.419812    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8" podUID="4edabf76-33a1-495b-a1a7-c5c36c0b3929" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 18:10:54.419299 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:10:54.419252    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8" podUID="4edabf76-33a1-495b-a1a7-c5c36c0b3929" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 18:11:04.419524 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:04.419474    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8" podUID="4edabf76-33a1-495b-a1a7-c5c36c0b3929" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 18:11:14.420205 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:14.420173    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8"
Sep 17 18:11:30.549513 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:30.549479    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8"]
Sep 17 18:11:30.549911 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:30.549732    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8" podUID="4edabf76-33a1-495b-a1a7-c5c36c0b3929" containerName="kserve-container" containerID="cri-o://284df10069fc76e87b52c91fde0f3c5a0dd6f9115fb82cd196277deec0e0bf9d" gracePeriod=30
Sep 17 18:11:30.549911 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:30.549758    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8" podUID="4edabf76-33a1-495b-a1a7-c5c36c0b3929" containerName="kube-rbac-proxy" containerID="cri-o://eb832e09058afb1eaad800aa60cc90118111d8904a122af43b4c2cdf6b30e6f0" gracePeriod=30
Sep 17 18:11:30.811401 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:30.811312    2581 generic.go:356] "Generic (PLEG): container finished" podID="4edabf76-33a1-495b-a1a7-c5c36c0b3929" containerID="eb832e09058afb1eaad800aa60cc90118111d8904a122af43b4c2cdf6b30e6f0" exitCode=2
Sep 17 18:11:30.811401 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:30.811380    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8" event={"ID":"4edabf76-33a1-495b-a1a7-c5c36c0b3929","Type":"ContainerDied","Data":"eb832e09058afb1eaad800aa60cc90118111d8904a122af43b4c2cdf6b30e6f0"}
Sep 17 18:11:34.415202 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:34.415134    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8" podUID="4edabf76-33a1-495b-a1a7-c5c36c0b3929" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 17 18:11:34.419443 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:34.419413    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8" podUID="4edabf76-33a1-495b-a1a7-c5c36c0b3929" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 18:11:34.988267 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:34.988246    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8"
Sep 17 18:11:35.130937 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:35.130902    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4edabf76-33a1-495b-a1a7-c5c36c0b3929-cabundle-cert\") pod \"4edabf76-33a1-495b-a1a7-c5c36c0b3929\" (UID: \"4edabf76-33a1-495b-a1a7-c5c36c0b3929\") "
Sep 17 18:11:35.131126 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:35.130960    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4edabf76-33a1-495b-a1a7-c5c36c0b3929-proxy-tls\") pod \"4edabf76-33a1-495b-a1a7-c5c36c0b3929\" (UID: \"4edabf76-33a1-495b-a1a7-c5c36c0b3929\") "
Sep 17 18:11:35.131126 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:35.130996    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4edabf76-33a1-495b-a1a7-c5c36c0b3929-kserve-provision-location\") pod \"4edabf76-33a1-495b-a1a7-c5c36c0b3929\" (UID: \"4edabf76-33a1-495b-a1a7-c5c36c0b3929\") "
Sep 17 18:11:35.131126 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:35.131061    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4edabf76-33a1-495b-a1a7-c5c36c0b3929-kube-rbac-proxy-sar-config\") pod \"4edabf76-33a1-495b-a1a7-c5c36c0b3929\" (UID: \"4edabf76-33a1-495b-a1a7-c5c36c0b3929\") "
Sep 17 18:11:35.131126 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:35.131091    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pn8f2\" (UniqueName: \"kubernetes.io/projected/4edabf76-33a1-495b-a1a7-c5c36c0b3929-kube-api-access-pn8f2\") pod \"4edabf76-33a1-495b-a1a7-c5c36c0b3929\" (UID: \"4edabf76-33a1-495b-a1a7-c5c36c0b3929\") "
Sep 17 18:11:35.131412 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:35.131381    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4edabf76-33a1-495b-a1a7-c5c36c0b3929-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4edabf76-33a1-495b-a1a7-c5c36c0b3929" (UID: "4edabf76-33a1-495b-a1a7-c5c36c0b3929"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:11:35.131412 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:35.131396    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4edabf76-33a1-495b-a1a7-c5c36c0b3929-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4edabf76-33a1-495b-a1a7-c5c36c0b3929" (UID: "4edabf76-33a1-495b-a1a7-c5c36c0b3929"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:11:35.131535 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:35.131462    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4edabf76-33a1-495b-a1a7-c5c36c0b3929-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4edabf76-33a1-495b-a1a7-c5c36c0b3929" (UID: "4edabf76-33a1-495b-a1a7-c5c36c0b3929"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:11:35.133355 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:35.133324    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4edabf76-33a1-495b-a1a7-c5c36c0b3929-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4edabf76-33a1-495b-a1a7-c5c36c0b3929" (UID: "4edabf76-33a1-495b-a1a7-c5c36c0b3929"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:11:35.133452 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:35.133363    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4edabf76-33a1-495b-a1a7-c5c36c0b3929-kube-api-access-pn8f2" (OuterVolumeSpecName: "kube-api-access-pn8f2") pod "4edabf76-33a1-495b-a1a7-c5c36c0b3929" (UID: "4edabf76-33a1-495b-a1a7-c5c36c0b3929"). InnerVolumeSpecName "kube-api-access-pn8f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:11:35.232383 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:35.232345    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4edabf76-33a1-495b-a1a7-c5c36c0b3929-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:11:35.232383 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:35.232372    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn8f2\" (UniqueName: \"kubernetes.io/projected/4edabf76-33a1-495b-a1a7-c5c36c0b3929-kube-api-access-pn8f2\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:11:35.232383 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:35.232382    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4edabf76-33a1-495b-a1a7-c5c36c0b3929-cabundle-cert\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:11:35.232383 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:35.232392    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4edabf76-33a1-495b-a1a7-c5c36c0b3929-proxy-tls\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:11:35.232704 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:35.232400    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4edabf76-33a1-495b-a1a7-c5c36c0b3929-kserve-provision-location\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:11:35.833883 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:35.833849    2581 generic.go:356] "Generic (PLEG): container finished" podID="4edabf76-33a1-495b-a1a7-c5c36c0b3929" containerID="284df10069fc76e87b52c91fde0f3c5a0dd6f9115fb82cd196277deec0e0bf9d" exitCode=0
Sep 17 18:11:35.834373 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:35.833913    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8"
Sep 17 18:11:35.834373 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:35.833929    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8" event={"ID":"4edabf76-33a1-495b-a1a7-c5c36c0b3929","Type":"ContainerDied","Data":"284df10069fc76e87b52c91fde0f3c5a0dd6f9115fb82cd196277deec0e0bf9d"}
Sep 17 18:11:35.834373 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:35.834008    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8" event={"ID":"4edabf76-33a1-495b-a1a7-c5c36c0b3929","Type":"ContainerDied","Data":"b0d60783b655a78d98be2eff11fa246c0f23cc1ba59741013f3bcbb2b984574e"}
Sep 17 18:11:35.834373 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:35.834044    2581 scope.go:117] "RemoveContainer" containerID="eb832e09058afb1eaad800aa60cc90118111d8904a122af43b4c2cdf6b30e6f0"
Sep 17 18:11:35.843013 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:35.842998    2581 scope.go:117] "RemoveContainer" containerID="284df10069fc76e87b52c91fde0f3c5a0dd6f9115fb82cd196277deec0e0bf9d"
Sep 17 18:11:35.850375 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:35.850356    2581 scope.go:117] "RemoveContainer" containerID="731e62e4726b62f6b46f569a94704615715c00c98db637a4488a4d56e55c98e5"
Sep 17 18:11:35.855534 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:35.855514    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8"]
Sep 17 18:11:35.859104 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:35.859082    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-c8kb8"]
Sep 17 18:11:35.860975 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:35.860960    2581 scope.go:117] "RemoveContainer" containerID="eb832e09058afb1eaad800aa60cc90118111d8904a122af43b4c2cdf6b30e6f0"
Sep 17 18:11:35.861278 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:11:35.861260    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb832e09058afb1eaad800aa60cc90118111d8904a122af43b4c2cdf6b30e6f0\": container with ID starting with eb832e09058afb1eaad800aa60cc90118111d8904a122af43b4c2cdf6b30e6f0 not found: ID does not exist" containerID="eb832e09058afb1eaad800aa60cc90118111d8904a122af43b4c2cdf6b30e6f0"
Sep 17 18:11:35.861324 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:35.861288    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb832e09058afb1eaad800aa60cc90118111d8904a122af43b4c2cdf6b30e6f0"} err="failed to get container status \"eb832e09058afb1eaad800aa60cc90118111d8904a122af43b4c2cdf6b30e6f0\": rpc error: code = NotFound desc = could not find container \"eb832e09058afb1eaad800aa60cc90118111d8904a122af43b4c2cdf6b30e6f0\": container with ID starting with eb832e09058afb1eaad800aa60cc90118111d8904a122af43b4c2cdf6b30e6f0 not found: ID does not exist"
Sep 17 18:11:35.861324 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:35.861305    2581 scope.go:117] "RemoveContainer" containerID="284df10069fc76e87b52c91fde0f3c5a0dd6f9115fb82cd196277deec0e0bf9d"
Sep 17 18:11:35.861578 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:11:35.861561    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"284df10069fc76e87b52c91fde0f3c5a0dd6f9115fb82cd196277deec0e0bf9d\": container with ID starting with 284df10069fc76e87b52c91fde0f3c5a0dd6f9115fb82cd196277deec0e0bf9d not found: ID does not exist" containerID="284df10069fc76e87b52c91fde0f3c5a0dd6f9115fb82cd196277deec0e0bf9d"
Sep 17 18:11:35.861621 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:35.861591    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"284df10069fc76e87b52c91fde0f3c5a0dd6f9115fb82cd196277deec0e0bf9d"} err="failed to get container status \"284df10069fc76e87b52c91fde0f3c5a0dd6f9115fb82cd196277deec0e0bf9d\": rpc error: code = NotFound desc = could not find container \"284df10069fc76e87b52c91fde0f3c5a0dd6f9115fb82cd196277deec0e0bf9d\": container with ID starting with 284df10069fc76e87b52c91fde0f3c5a0dd6f9115fb82cd196277deec0e0bf9d not found: ID does not exist"
Sep 17 18:11:35.861621 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:35.861607    2581 scope.go:117] "RemoveContainer" containerID="731e62e4726b62f6b46f569a94704615715c00c98db637a4488a4d56e55c98e5"
Sep 17 18:11:35.861837 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:11:35.861824    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"731e62e4726b62f6b46f569a94704615715c00c98db637a4488a4d56e55c98e5\": container with ID starting with 731e62e4726b62f6b46f569a94704615715c00c98db637a4488a4d56e55c98e5 not found: ID does not exist" containerID="731e62e4726b62f6b46f569a94704615715c00c98db637a4488a4d56e55c98e5"
Sep 17 18:11:35.861897 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:35.861840    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"731e62e4726b62f6b46f569a94704615715c00c98db637a4488a4d56e55c98e5"} err="failed to get container status \"731e62e4726b62f6b46f569a94704615715c00c98db637a4488a4d56e55c98e5\": rpc error: code = NotFound desc = could not find container \"731e62e4726b62f6b46f569a94704615715c00c98db637a4488a4d56e55c98e5\": container with ID starting with 731e62e4726b62f6b46f569a94704615715c00c98db637a4488a4d56e55c98e5 not found: ID does not exist"
Sep 17 18:11:36.048443 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:36.048415    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4edabf76-33a1-495b-a1a7-c5c36c0b3929" path="/var/lib/kubelet/pods/4edabf76-33a1-495b-a1a7-c5c36c0b3929/volumes"
Sep 17 18:11:40.806400 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.806327    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8"]
Sep 17 18:11:40.806725 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.806633    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fa9a63c-679c-4161-b889-674a6a4c198e" containerName="kube-rbac-proxy"
Sep 17 18:11:40.806725 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.806646    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fa9a63c-679c-4161-b889-674a6a4c198e" containerName="kube-rbac-proxy"
Sep 17 18:11:40.806725 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.806656    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fa9a63c-679c-4161-b889-674a6a4c198e" containerName="kserve-container"
Sep 17 18:11:40.806725 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.806662    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fa9a63c-679c-4161-b889-674a6a4c198e" containerName="kserve-container"
Sep 17 18:11:40.806725 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.806672    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4edabf76-33a1-495b-a1a7-c5c36c0b3929" containerName="storage-initializer"
Sep 17 18:11:40.806725 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.806677    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4edabf76-33a1-495b-a1a7-c5c36c0b3929" containerName="storage-initializer"
Sep 17 18:11:40.806725 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.806682    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4edabf76-33a1-495b-a1a7-c5c36c0b3929" containerName="kube-rbac-proxy"
Sep 17 18:11:40.806725 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.806688    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4edabf76-33a1-495b-a1a7-c5c36c0b3929" containerName="kube-rbac-proxy"
Sep 17 18:11:40.806725 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.806695    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fa9a63c-679c-4161-b889-674a6a4c198e" containerName="storage-initializer"
Sep 17 18:11:40.806725 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.806701    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fa9a63c-679c-4161-b889-674a6a4c198e" containerName="storage-initializer"
Sep 17 18:11:40.806725 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.806708    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bd93e17-0115-4979-883a-aef312aa68b3" containerName="storage-initializer"
Sep 17 18:11:40.806725 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.806714    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bd93e17-0115-4979-883a-aef312aa68b3" containerName="storage-initializer"
Sep 17 18:11:40.806725 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.806722    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bd93e17-0115-4979-883a-aef312aa68b3" containerName="kserve-container"
Sep 17 18:11:40.806725 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.806728    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bd93e17-0115-4979-883a-aef312aa68b3" containerName="kserve-container"
Sep 17 18:11:40.807178 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.806735    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4edabf76-33a1-495b-a1a7-c5c36c0b3929" containerName="kserve-container"
Sep 17 18:11:40.807178 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.806740    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4edabf76-33a1-495b-a1a7-c5c36c0b3929" containerName="kserve-container"
Sep 17 18:11:40.807178 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.806752    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bd93e17-0115-4979-883a-aef312aa68b3" containerName="kube-rbac-proxy"
Sep 17 18:11:40.807178 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.806758    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bd93e17-0115-4979-883a-aef312aa68b3" containerName="kube-rbac-proxy"
Sep 17 18:11:40.807178 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.806821    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fa9a63c-679c-4161-b889-674a6a4c198e" containerName="kserve-container"
Sep 17 18:11:40.807178 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.806829    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fa9a63c-679c-4161-b889-674a6a4c198e" containerName="kube-rbac-proxy"
Sep 17 18:11:40.807178 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.806835    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bd93e17-0115-4979-883a-aef312aa68b3" containerName="kube-rbac-proxy"
Sep 17 18:11:40.807178 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.806841    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4edabf76-33a1-495b-a1a7-c5c36c0b3929" containerName="kserve-container"
Sep 17 18:11:40.807178 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.806848    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bd93e17-0115-4979-883a-aef312aa68b3" containerName="kserve-container"
Sep 17 18:11:40.807178 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.806855    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4edabf76-33a1-495b-a1a7-c5c36c0b3929" containerName="kube-rbac-proxy"
Sep 17 18:11:40.810228 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.810213    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8"
Sep 17 18:11:40.813649 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.813628    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:11:40.813918 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.813899    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:11:40.814188 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.814144    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:11:40.814301 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.814170    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:11:40.814301 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.814213    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-c81569-kube-rbac-proxy-sar-config\""
Sep 17 18:11:40.814301 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.814204    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-c81569-predictor-serving-cert\""
Sep 17 18:11:40.814495 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.814482    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w6q8k\""
Sep 17 18:11:40.823521 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.823492    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8"]
Sep 17 18:11:40.879426 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.879401    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d04c05e-6542-4f77-b794-7ee1af0be89a-kube-rbac-proxy-sar-config\") pod \"isvc-primary-c81569-predictor-d8cc8857-btdk8\" (UID: \"2d04c05e-6542-4f77-b794-7ee1af0be89a\") " pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8"
Sep 17 18:11:40.879526 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.879443    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d04c05e-6542-4f77-b794-7ee1af0be89a-proxy-tls\") pod \"isvc-primary-c81569-predictor-d8cc8857-btdk8\" (UID: \"2d04c05e-6542-4f77-b794-7ee1af0be89a\") " pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8"
Sep 17 18:11:40.879526 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.879505    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk6b9\" (UniqueName: \"kubernetes.io/projected/2d04c05e-6542-4f77-b794-7ee1af0be89a-kube-api-access-vk6b9\") pod \"isvc-primary-c81569-predictor-d8cc8857-btdk8\" (UID: \"2d04c05e-6542-4f77-b794-7ee1af0be89a\") " pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8"
Sep 17 18:11:40.879637 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.879524    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d04c05e-6542-4f77-b794-7ee1af0be89a-cabundle-cert\") pod \"isvc-primary-c81569-predictor-d8cc8857-btdk8\" (UID: \"2d04c05e-6542-4f77-b794-7ee1af0be89a\") " pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8"
Sep 17 18:11:40.879637 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.879548    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d04c05e-6542-4f77-b794-7ee1af0be89a-kserve-provision-location\") pod \"isvc-primary-c81569-predictor-d8cc8857-btdk8\" (UID: \"2d04c05e-6542-4f77-b794-7ee1af0be89a\") " pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8"
Sep 17 18:11:40.979945 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.979910    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk6b9\" (UniqueName: \"kubernetes.io/projected/2d04c05e-6542-4f77-b794-7ee1af0be89a-kube-api-access-vk6b9\") pod \"isvc-primary-c81569-predictor-d8cc8857-btdk8\" (UID: \"2d04c05e-6542-4f77-b794-7ee1af0be89a\") " pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8"
Sep 17 18:11:40.979945 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.979943    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d04c05e-6542-4f77-b794-7ee1af0be89a-cabundle-cert\") pod \"isvc-primary-c81569-predictor-d8cc8857-btdk8\" (UID: \"2d04c05e-6542-4f77-b794-7ee1af0be89a\") " pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8"
Sep 17 18:11:40.980209 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.979966    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d04c05e-6542-4f77-b794-7ee1af0be89a-kserve-provision-location\") pod \"isvc-primary-c81569-predictor-d8cc8857-btdk8\" (UID: \"2d04c05e-6542-4f77-b794-7ee1af0be89a\") " pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8"
Sep 17 18:11:40.980209 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.980018    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d04c05e-6542-4f77-b794-7ee1af0be89a-kube-rbac-proxy-sar-config\") pod \"isvc-primary-c81569-predictor-d8cc8857-btdk8\" (UID: \"2d04c05e-6542-4f77-b794-7ee1af0be89a\") " pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8"
Sep 17 18:11:40.980209 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.980051    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d04c05e-6542-4f77-b794-7ee1af0be89a-proxy-tls\") pod \"isvc-primary-c81569-predictor-d8cc8857-btdk8\" (UID: \"2d04c05e-6542-4f77-b794-7ee1af0be89a\") " pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8"
Sep 17 18:11:40.980209 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:11:40.980185    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-c81569-predictor-serving-cert: secret "isvc-primary-c81569-predictor-serving-cert" not found
Sep 17 18:11:40.980409 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:11:40.980256    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d04c05e-6542-4f77-b794-7ee1af0be89a-proxy-tls podName:2d04c05e-6542-4f77-b794-7ee1af0be89a nodeName:}" failed. No retries permitted until 2026-09-17 18:11:41.480235275 +0000 UTC m=+883.580699092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2d04c05e-6542-4f77-b794-7ee1af0be89a-proxy-tls") pod "isvc-primary-c81569-predictor-d8cc8857-btdk8" (UID: "2d04c05e-6542-4f77-b794-7ee1af0be89a") : secret "isvc-primary-c81569-predictor-serving-cert" not found
Sep 17 18:11:40.980471 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.980404    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d04c05e-6542-4f77-b794-7ee1af0be89a-kserve-provision-location\") pod \"isvc-primary-c81569-predictor-d8cc8857-btdk8\" (UID: \"2d04c05e-6542-4f77-b794-7ee1af0be89a\") " pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8"
Sep 17 18:11:40.980669 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.980652    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d04c05e-6542-4f77-b794-7ee1af0be89a-cabundle-cert\") pod \"isvc-primary-c81569-predictor-d8cc8857-btdk8\" (UID: \"2d04c05e-6542-4f77-b794-7ee1af0be89a\") " pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8"
Sep 17 18:11:40.980713 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.980686    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d04c05e-6542-4f77-b794-7ee1af0be89a-kube-rbac-proxy-sar-config\") pod \"isvc-primary-c81569-predictor-d8cc8857-btdk8\" (UID: \"2d04c05e-6542-4f77-b794-7ee1af0be89a\") " pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8"
Sep 17 18:11:40.994857 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:40.994823    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk6b9\" (UniqueName: \"kubernetes.io/projected/2d04c05e-6542-4f77-b794-7ee1af0be89a-kube-api-access-vk6b9\") pod \"isvc-primary-c81569-predictor-d8cc8857-btdk8\" (UID: \"2d04c05e-6542-4f77-b794-7ee1af0be89a\") " pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8"
Sep 17 18:11:41.484555 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:41.484518    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d04c05e-6542-4f77-b794-7ee1af0be89a-proxy-tls\") pod \"isvc-primary-c81569-predictor-d8cc8857-btdk8\" (UID: \"2d04c05e-6542-4f77-b794-7ee1af0be89a\") " pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8"
Sep 17 18:11:41.487114 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:41.487091    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d04c05e-6542-4f77-b794-7ee1af0be89a-proxy-tls\") pod \"isvc-primary-c81569-predictor-d8cc8857-btdk8\" (UID: \"2d04c05e-6542-4f77-b794-7ee1af0be89a\") " pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8"
Sep 17 18:11:41.723247 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:41.723209    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8"
Sep 17 18:11:42.005375 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:42.005336    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8"]
Sep 17 18:11:42.009402 ip-10-0-134-209 kubenswrapper[2581]: W0917 18:11:42.009371    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d04c05e_6542_4f77_b794_7ee1af0be89a.slice/crio-9f9cf7fc493c9ea256a262e5fbe25a602e777b6362547df0a24c5d19640ba6fe WatchSource:0}: Error finding container 9f9cf7fc493c9ea256a262e5fbe25a602e777b6362547df0a24c5d19640ba6fe: Status 404 returned error can't find the container with id 9f9cf7fc493c9ea256a262e5fbe25a602e777b6362547df0a24c5d19640ba6fe
Sep 17 18:11:42.863811 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:42.863771    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8" event={"ID":"2d04c05e-6542-4f77-b794-7ee1af0be89a","Type":"ContainerStarted","Data":"1432fbc68462c1b20a7452fb9118dc82c206434f3b535d102c577b1db126902f"}
Sep 17 18:11:42.864000 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:42.863831    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8" event={"ID":"2d04c05e-6542-4f77-b794-7ee1af0be89a","Type":"ContainerStarted","Data":"9f9cf7fc493c9ea256a262e5fbe25a602e777b6362547df0a24c5d19640ba6fe"}
Sep 17 18:11:46.882115 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:46.882075    2581 generic.go:356] "Generic (PLEG): container finished" podID="2d04c05e-6542-4f77-b794-7ee1af0be89a" containerID="1432fbc68462c1b20a7452fb9118dc82c206434f3b535d102c577b1db126902f" exitCode=0
Sep 17 18:11:46.882498 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:46.882166    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8" event={"ID":"2d04c05e-6542-4f77-b794-7ee1af0be89a","Type":"ContainerDied","Data":"1432fbc68462c1b20a7452fb9118dc82c206434f3b535d102c577b1db126902f"}
Sep 17 18:11:47.889116 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:47.889079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8" event={"ID":"2d04c05e-6542-4f77-b794-7ee1af0be89a","Type":"ContainerStarted","Data":"3c7c3ddba00e5dd7bee7b2687f47ed71ae55ace00322b68ea76eacd2fa390394"}
Sep 17 18:11:47.889551 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:47.889138    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8" event={"ID":"2d04c05e-6542-4f77-b794-7ee1af0be89a","Type":"ContainerStarted","Data":"f7832f6dcaea4b7ceb17b857bb822e18ff2f68b5429915b1b55077dd98a4abca"}
Sep 17 18:11:47.889551 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:47.889452    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8"
Sep 17 18:11:47.889670 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:47.889595    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8"
Sep 17 18:11:47.890569 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:47.890545    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8" podUID="2d04c05e-6542-4f77-b794-7ee1af0be89a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 18:11:47.907943 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:47.907905    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8" podStartSLOduration=4.087587407 podStartE2EDuration="7.907892245s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.820304838s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:11:47.907067589 +0000 UTC m=+890.007531409" watchObservedRunningTime="2026-09-17 18:11:47.907892245 +0000 UTC m=+890.008356068"
Sep 17 18:11:48.893454 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:48.893418    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8" podUID="2d04c05e-6542-4f77-b794-7ee1af0be89a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 18:11:53.897892 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:53.897863    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8"
Sep 17 18:11:53.898372 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:53.898348    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8" podUID="2d04c05e-6542-4f77-b794-7ee1af0be89a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 18:11:59.965526 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:59.965500    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm8hx_913ce1e9-9fb6-4049-a06d-9ebc69d3e528/ovn-acl-logging/0.log"
Sep 17 18:11:59.966472 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:11:59.966452    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm8hx_913ce1e9-9fb6-4049-a06d-9ebc69d3e528/ovn-acl-logging/0.log"
Sep 17 18:12:03.899324 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:12:03.899279    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8" podUID="2d04c05e-6542-4f77-b794-7ee1af0be89a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 18:12:13.899288 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:12:13.899245    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8" podUID="2d04c05e-6542-4f77-b794-7ee1af0be89a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 18:12:23.899328 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:12:23.899289    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8" podUID="2d04c05e-6542-4f77-b794-7ee1af0be89a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 18:12:33.898758 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:12:33.898722    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8" podUID="2d04c05e-6542-4f77-b794-7ee1af0be89a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 18:12:43.899052 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:12:43.899014    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8" podUID="2d04c05e-6542-4f77-b794-7ee1af0be89a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 18:12:53.898702 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:12:53.898661    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8" podUID="2d04c05e-6542-4f77-b794-7ee1af0be89a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 18:13:03.899334 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:03.899305    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8"
Sep 17 18:13:25.964532 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:25.964495    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9"]
Sep 17 18:13:25.968043 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:25.968019    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9"
Sep 17 18:13:25.969803 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:25.969780    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-c81569-predictor-serving-cert\""
Sep 17 18:13:25.969932 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:25.969818    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-c81569-dockercfg-2xk8x\""
Sep 17 18:13:25.969932 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:25.969887    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-c81569\""
Sep 17 18:13:25.970096 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:25.970083    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-c81569-kube-rbac-proxy-sar-config\""
Sep 17 18:13:25.979441 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:25.979422    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9"]
Sep 17 18:13:26.080457 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:26.080419    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce7faf98-4168-4b83-8889-d5499f2b0a9d-cabundle-cert\") pod \"isvc-secondary-c81569-predictor-9b69cb5d-q5cn9\" (UID: \"ce7faf98-4168-4b83-8889-d5499f2b0a9d\") " pod="kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9"
Sep 17 18:13:26.080457 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:26.080464    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce7faf98-4168-4b83-8889-d5499f2b0a9d-kserve-provision-location\") pod \"isvc-secondary-c81569-predictor-9b69cb5d-q5cn9\" (UID: \"ce7faf98-4168-4b83-8889-d5499f2b0a9d\") " pod="kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9"
Sep 17 18:13:26.080635 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:26.080502    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce7faf98-4168-4b83-8889-d5499f2b0a9d-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-c81569-predictor-9b69cb5d-q5cn9\" (UID: \"ce7faf98-4168-4b83-8889-d5499f2b0a9d\") " pod="kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9"
Sep 17 18:13:26.080635 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:26.080531    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce7faf98-4168-4b83-8889-d5499f2b0a9d-proxy-tls\") pod \"isvc-secondary-c81569-predictor-9b69cb5d-q5cn9\" (UID: \"ce7faf98-4168-4b83-8889-d5499f2b0a9d\") " pod="kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9"
Sep 17 18:13:26.080635 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:26.080545    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbmxd\" (UniqueName: \"kubernetes.io/projected/ce7faf98-4168-4b83-8889-d5499f2b0a9d-kube-api-access-hbmxd\") pod \"isvc-secondary-c81569-predictor-9b69cb5d-q5cn9\" (UID: \"ce7faf98-4168-4b83-8889-d5499f2b0a9d\") " pod="kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9"
Sep 17 18:13:26.181690 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:26.181664    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce7faf98-4168-4b83-8889-d5499f2b0a9d-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-c81569-predictor-9b69cb5d-q5cn9\" (UID: \"ce7faf98-4168-4b83-8889-d5499f2b0a9d\") " pod="kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9"
Sep 17 18:13:26.181826 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:26.181702    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce7faf98-4168-4b83-8889-d5499f2b0a9d-proxy-tls\") pod \"isvc-secondary-c81569-predictor-9b69cb5d-q5cn9\" (UID: \"ce7faf98-4168-4b83-8889-d5499f2b0a9d\") " pod="kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9"
Sep 17 18:13:26.181826 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:26.181719    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbmxd\" (UniqueName: \"kubernetes.io/projected/ce7faf98-4168-4b83-8889-d5499f2b0a9d-kube-api-access-hbmxd\") pod \"isvc-secondary-c81569-predictor-9b69cb5d-q5cn9\" (UID: \"ce7faf98-4168-4b83-8889-d5499f2b0a9d\") " pod="kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9"
Sep 17 18:13:26.181943 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:26.181832    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce7faf98-4168-4b83-8889-d5499f2b0a9d-cabundle-cert\") pod \"isvc-secondary-c81569-predictor-9b69cb5d-q5cn9\" (UID: \"ce7faf98-4168-4b83-8889-d5499f2b0a9d\") " pod="kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9"
Sep 17 18:13:26.181943 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:26.181870    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce7faf98-4168-4b83-8889-d5499f2b0a9d-kserve-provision-location\") pod \"isvc-secondary-c81569-predictor-9b69cb5d-q5cn9\" (UID: \"ce7faf98-4168-4b83-8889-d5499f2b0a9d\") " pod="kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9"
Sep 17 18:13:26.181943 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:13:26.181915    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-c81569-predictor-serving-cert: secret "isvc-secondary-c81569-predictor-serving-cert" not found
Sep 17 18:13:26.182109 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:13:26.181985    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce7faf98-4168-4b83-8889-d5499f2b0a9d-proxy-tls podName:ce7faf98-4168-4b83-8889-d5499f2b0a9d nodeName:}" failed. No retries permitted until 2026-09-17 18:13:26.681967463 +0000 UTC m=+988.782431276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ce7faf98-4168-4b83-8889-d5499f2b0a9d-proxy-tls") pod "isvc-secondary-c81569-predictor-9b69cb5d-q5cn9" (UID: "ce7faf98-4168-4b83-8889-d5499f2b0a9d") : secret "isvc-secondary-c81569-predictor-serving-cert" not found
Sep 17 18:13:26.182301 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:26.182276    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce7faf98-4168-4b83-8889-d5499f2b0a9d-kserve-provision-location\") pod \"isvc-secondary-c81569-predictor-9b69cb5d-q5cn9\" (UID: \"ce7faf98-4168-4b83-8889-d5499f2b0a9d\") " pod="kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9"
Sep 17 18:13:26.182494 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:26.182469    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce7faf98-4168-4b83-8889-d5499f2b0a9d-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-c81569-predictor-9b69cb5d-q5cn9\" (UID: \"ce7faf98-4168-4b83-8889-d5499f2b0a9d\") " pod="kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9"
Sep 17 18:13:26.182608 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:26.182571    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce7faf98-4168-4b83-8889-d5499f2b0a9d-cabundle-cert\") pod \"isvc-secondary-c81569-predictor-9b69cb5d-q5cn9\" (UID: \"ce7faf98-4168-4b83-8889-d5499f2b0a9d\") " pod="kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9"
Sep 17 18:13:26.190069 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:26.190040    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbmxd\" (UniqueName: \"kubernetes.io/projected/ce7faf98-4168-4b83-8889-d5499f2b0a9d-kube-api-access-hbmxd\") pod \"isvc-secondary-c81569-predictor-9b69cb5d-q5cn9\" (UID: \"ce7faf98-4168-4b83-8889-d5499f2b0a9d\") " pod="kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9"
Sep 17 18:13:26.687195 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:26.687145    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce7faf98-4168-4b83-8889-d5499f2b0a9d-proxy-tls\") pod \"isvc-secondary-c81569-predictor-9b69cb5d-q5cn9\" (UID: \"ce7faf98-4168-4b83-8889-d5499f2b0a9d\") " pod="kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9"
Sep 17 18:13:26.689671 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:26.689650    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce7faf98-4168-4b83-8889-d5499f2b0a9d-proxy-tls\") pod \"isvc-secondary-c81569-predictor-9b69cb5d-q5cn9\" (UID: \"ce7faf98-4168-4b83-8889-d5499f2b0a9d\") " pod="kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9"
Sep 17 18:13:26.881294 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:26.881258    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9"
Sep 17 18:13:27.161383 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:27.161361    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9"]
Sep 17 18:13:27.164059 ip-10-0-134-209 kubenswrapper[2581]: W0917 18:13:27.164020    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce7faf98_4168_4b83_8889_d5499f2b0a9d.slice/crio-691bfa29641f6878f68b05cb81989cd15482bed8ec17a4d2ca54b9f6fdbb1f03 WatchSource:0}: Error finding container 691bfa29641f6878f68b05cb81989cd15482bed8ec17a4d2ca54b9f6fdbb1f03: Status 404 returned error can't find the container with id 691bfa29641f6878f68b05cb81989cd15482bed8ec17a4d2ca54b9f6fdbb1f03
Sep 17 18:13:27.289549 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:27.289511    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9" event={"ID":"ce7faf98-4168-4b83-8889-d5499f2b0a9d","Type":"ContainerStarted","Data":"81d0a48090450b4de6c712ce20898de1760fa4f91ff9fe34f2b35d574af51e28"}
Sep 17 18:13:27.289669 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:27.289565    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9" event={"ID":"ce7faf98-4168-4b83-8889-d5499f2b0a9d","Type":"ContainerStarted","Data":"691bfa29641f6878f68b05cb81989cd15482bed8ec17a4d2ca54b9f6fdbb1f03"}
Sep 17 18:13:33.316296 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:33.316269    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c81569-predictor-9b69cb5d-q5cn9_ce7faf98-4168-4b83-8889-d5499f2b0a9d/storage-initializer/0.log"
Sep 17 18:13:33.316628 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:33.316312    2581 generic.go:356] "Generic (PLEG): container finished" podID="ce7faf98-4168-4b83-8889-d5499f2b0a9d" containerID="81d0a48090450b4de6c712ce20898de1760fa4f91ff9fe34f2b35d574af51e28" exitCode=1
Sep 17 18:13:33.316628 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:33.316356    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9" event={"ID":"ce7faf98-4168-4b83-8889-d5499f2b0a9d","Type":"ContainerDied","Data":"81d0a48090450b4de6c712ce20898de1760fa4f91ff9fe34f2b35d574af51e28"}
Sep 17 18:13:34.322535 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:34.322508    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c81569-predictor-9b69cb5d-q5cn9_ce7faf98-4168-4b83-8889-d5499f2b0a9d/storage-initializer/0.log"
Sep 17 18:13:34.322902 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:34.322605    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9" event={"ID":"ce7faf98-4168-4b83-8889-d5499f2b0a9d","Type":"ContainerStarted","Data":"57b53e99e0cd23535b544b031ce12980ca946bd831db4d54cb388063a7e7da9b"}
Sep 17 18:13:40.347420 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:40.347391    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c81569-predictor-9b69cb5d-q5cn9_ce7faf98-4168-4b83-8889-d5499f2b0a9d/storage-initializer/1.log"
Sep 17 18:13:40.347865 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:40.347804    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c81569-predictor-9b69cb5d-q5cn9_ce7faf98-4168-4b83-8889-d5499f2b0a9d/storage-initializer/0.log"
Sep 17 18:13:40.347865 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:40.347843    2581 generic.go:356] "Generic (PLEG): container finished" podID="ce7faf98-4168-4b83-8889-d5499f2b0a9d" containerID="57b53e99e0cd23535b544b031ce12980ca946bd831db4d54cb388063a7e7da9b" exitCode=1
Sep 17 18:13:40.347972 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:40.347911    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9" event={"ID":"ce7faf98-4168-4b83-8889-d5499f2b0a9d","Type":"ContainerDied","Data":"57b53e99e0cd23535b544b031ce12980ca946bd831db4d54cb388063a7e7da9b"}
Sep 17 18:13:40.348019 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:40.348000    2581 scope.go:117] "RemoveContainer" containerID="81d0a48090450b4de6c712ce20898de1760fa4f91ff9fe34f2b35d574af51e28"
Sep 17 18:13:40.348390 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:40.348370    2581 scope.go:117] "RemoveContainer" containerID="81d0a48090450b4de6c712ce20898de1760fa4f91ff9fe34f2b35d574af51e28"
Sep 17 18:13:40.360585 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:13:40.360563    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-c81569-predictor-9b69cb5d-q5cn9_kserve-ci-e2e-test(ce7faf98-4168-4b83-8889-d5499f2b0a9d)\"" pod="kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9" podUID="ce7faf98-4168-4b83-8889-d5499f2b0a9d"
Sep 17 18:13:40.360788 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:13:40.360764    2581 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-c81569-predictor-9b69cb5d-q5cn9_kserve-ci-e2e-test_ce7faf98-4168-4b83-8889-d5499f2b0a9d_0 in pod sandbox 691bfa29641f6878f68b05cb81989cd15482bed8ec17a4d2ca54b9f6fdbb1f03 from index: no such id: '81d0a48090450b4de6c712ce20898de1760fa4f91ff9fe34f2b35d574af51e28'" containerID="81d0a48090450b4de6c712ce20898de1760fa4f91ff9fe34f2b35d574af51e28"
Sep 17 18:13:40.360870 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:40.360798    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81d0a48090450b4de6c712ce20898de1760fa4f91ff9fe34f2b35d574af51e28"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-c81569-predictor-9b69cb5d-q5cn9_kserve-ci-e2e-test_ce7faf98-4168-4b83-8889-d5499f2b0a9d_0 in pod sandbox 691bfa29641f6878f68b05cb81989cd15482bed8ec17a4d2ca54b9f6fdbb1f03 from index: no such id: '81d0a48090450b4de6c712ce20898de1760fa4f91ff9fe34f2b35d574af51e28'"
Sep 17 18:13:41.353562 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:41.353532    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c81569-predictor-9b69cb5d-q5cn9_ce7faf98-4168-4b83-8889-d5499f2b0a9d/storage-initializer/1.log"
Sep 17 18:13:43.977960 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:43.977928    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9"]
Sep 17 18:13:44.007667 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:44.007635    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8"]
Sep 17 18:13:44.007933 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:44.007892    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8" podUID="2d04c05e-6542-4f77-b794-7ee1af0be89a" containerName="kserve-container" containerID="cri-o://f7832f6dcaea4b7ceb17b857bb822e18ff2f68b5429915b1b55077dd98a4abca" gracePeriod=30
Sep 17 18:13:44.008035 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:44.007954    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8" podUID="2d04c05e-6542-4f77-b794-7ee1af0be89a" containerName="kube-rbac-proxy" containerID="cri-o://3c7c3ddba00e5dd7bee7b2687f47ed71ae55ace00322b68ea76eacd2fa390394" gracePeriod=30
Sep 17 18:13:44.128399 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:44.128375    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c81569-predictor-9b69cb5d-q5cn9_ce7faf98-4168-4b83-8889-d5499f2b0a9d/storage-initializer/1.log"
Sep 17 18:13:44.130680 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:44.130664    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9"
Sep 17 18:13:44.238935 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:44.238876    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce7faf98-4168-4b83-8889-d5499f2b0a9d-cabundle-cert\") pod \"ce7faf98-4168-4b83-8889-d5499f2b0a9d\" (UID: \"ce7faf98-4168-4b83-8889-d5499f2b0a9d\") "
Sep 17 18:13:44.238935 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:44.238914    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce7faf98-4168-4b83-8889-d5499f2b0a9d-kserve-provision-location\") pod \"ce7faf98-4168-4b83-8889-d5499f2b0a9d\" (UID: \"ce7faf98-4168-4b83-8889-d5499f2b0a9d\") "
Sep 17 18:13:44.239216 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:44.238943    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbmxd\" (UniqueName: \"kubernetes.io/projected/ce7faf98-4168-4b83-8889-d5499f2b0a9d-kube-api-access-hbmxd\") pod \"ce7faf98-4168-4b83-8889-d5499f2b0a9d\" (UID: \"ce7faf98-4168-4b83-8889-d5499f2b0a9d\") "
Sep 17 18:13:44.239216 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:44.239063    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce7faf98-4168-4b83-8889-d5499f2b0a9d-kube-rbac-proxy-sar-config\") pod \"ce7faf98-4168-4b83-8889-d5499f2b0a9d\" (UID: \"ce7faf98-4168-4b83-8889-d5499f2b0a9d\") "
Sep 17 18:13:44.239216 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:44.239103    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce7faf98-4168-4b83-8889-d5499f2b0a9d-proxy-tls\") pod \"ce7faf98-4168-4b83-8889-d5499f2b0a9d\" (UID: \"ce7faf98-4168-4b83-8889-d5499f2b0a9d\") "
Sep 17 18:13:44.239348 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:44.239208    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce7faf98-4168-4b83-8889-d5499f2b0a9d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ce7faf98-4168-4b83-8889-d5499f2b0a9d" (UID: "ce7faf98-4168-4b83-8889-d5499f2b0a9d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:13:44.239348 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:44.239234    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce7faf98-4168-4b83-8889-d5499f2b0a9d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ce7faf98-4168-4b83-8889-d5499f2b0a9d" (UID: "ce7faf98-4168-4b83-8889-d5499f2b0a9d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:13:44.239435 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:44.239401    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce7faf98-4168-4b83-8889-d5499f2b0a9d-cabundle-cert\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:13:44.239435 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:44.239422    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce7faf98-4168-4b83-8889-d5499f2b0a9d-kserve-provision-location\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:13:44.239496 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:44.239450    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce7faf98-4168-4b83-8889-d5499f2b0a9d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ce7faf98-4168-4b83-8889-d5499f2b0a9d" (UID: "ce7faf98-4168-4b83-8889-d5499f2b0a9d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:13:44.241302 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:44.241274    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce7faf98-4168-4b83-8889-d5499f2b0a9d-kube-api-access-hbmxd" (OuterVolumeSpecName: "kube-api-access-hbmxd") pod "ce7faf98-4168-4b83-8889-d5499f2b0a9d" (UID: "ce7faf98-4168-4b83-8889-d5499f2b0a9d"). InnerVolumeSpecName "kube-api-access-hbmxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:13:44.241302 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:44.241280    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce7faf98-4168-4b83-8889-d5499f2b0a9d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ce7faf98-4168-4b83-8889-d5499f2b0a9d" (UID: "ce7faf98-4168-4b83-8889-d5499f2b0a9d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:13:44.340359 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:44.340334    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbmxd\" (UniqueName: \"kubernetes.io/projected/ce7faf98-4168-4b83-8889-d5499f2b0a9d-kube-api-access-hbmxd\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:13:44.340359 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:44.340354    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce7faf98-4168-4b83-8889-d5499f2b0a9d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:13:44.340494 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:44.340363    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce7faf98-4168-4b83-8889-d5499f2b0a9d-proxy-tls\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:13:44.367690 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:44.367668    2581 generic.go:356] "Generic (PLEG): container finished" podID="2d04c05e-6542-4f77-b794-7ee1af0be89a" containerID="3c7c3ddba00e5dd7bee7b2687f47ed71ae55ace00322b68ea76eacd2fa390394" exitCode=2
Sep 17 18:13:44.367785 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:44.367749    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8" event={"ID":"2d04c05e-6542-4f77-b794-7ee1af0be89a","Type":"ContainerDied","Data":"3c7c3ddba00e5dd7bee7b2687f47ed71ae55ace00322b68ea76eacd2fa390394"}
Sep 17 18:13:44.368999 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:44.368985    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c81569-predictor-9b69cb5d-q5cn9_ce7faf98-4168-4b83-8889-d5499f2b0a9d/storage-initializer/1.log"
Sep 17 18:13:44.369104 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:44.369087    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9"
Sep 17 18:13:44.369192 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:44.369099    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9" event={"ID":"ce7faf98-4168-4b83-8889-d5499f2b0a9d","Type":"ContainerDied","Data":"691bfa29641f6878f68b05cb81989cd15482bed8ec17a4d2ca54b9f6fdbb1f03"}
Sep 17 18:13:44.369192 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:44.369176    2581 scope.go:117] "RemoveContainer" containerID="57b53e99e0cd23535b544b031ce12980ca946bd831db4d54cb388063a7e7da9b"
Sep 17 18:13:44.402692 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:44.402670    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9"]
Sep 17 18:13:44.405162 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:44.405128    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c81569-predictor-9b69cb5d-q5cn9"]
Sep 17 18:13:46.049744 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:46.049707    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce7faf98-4168-4b83-8889-d5499f2b0a9d" path="/var/lib/kubelet/pods/ce7faf98-4168-4b83-8889-d5499f2b0a9d/volumes"
Sep 17 18:13:48.543425 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:48.543398    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8"
Sep 17 18:13:48.679408 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:48.679387    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d04c05e-6542-4f77-b794-7ee1af0be89a-proxy-tls\") pod \"2d04c05e-6542-4f77-b794-7ee1af0be89a\" (UID: \"2d04c05e-6542-4f77-b794-7ee1af0be89a\") "
Sep 17 18:13:48.679553 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:48.679470    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d04c05e-6542-4f77-b794-7ee1af0be89a-kube-rbac-proxy-sar-config\") pod \"2d04c05e-6542-4f77-b794-7ee1af0be89a\" (UID: \"2d04c05e-6542-4f77-b794-7ee1af0be89a\") "
Sep 17 18:13:48.679553 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:48.679519    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vk6b9\" (UniqueName: \"kubernetes.io/projected/2d04c05e-6542-4f77-b794-7ee1af0be89a-kube-api-access-vk6b9\") pod \"2d04c05e-6542-4f77-b794-7ee1af0be89a\" (UID: \"2d04c05e-6542-4f77-b794-7ee1af0be89a\") "
Sep 17 18:13:48.679553 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:48.679537    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d04c05e-6542-4f77-b794-7ee1af0be89a-cabundle-cert\") pod \"2d04c05e-6542-4f77-b794-7ee1af0be89a\" (UID: \"2d04c05e-6542-4f77-b794-7ee1af0be89a\") "
Sep 17 18:13:48.679727 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:48.679561    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d04c05e-6542-4f77-b794-7ee1af0be89a-kserve-provision-location\") pod \"2d04c05e-6542-4f77-b794-7ee1af0be89a\" (UID: \"2d04c05e-6542-4f77-b794-7ee1af0be89a\") "
Sep 17 18:13:48.679903 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:48.679877    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d04c05e-6542-4f77-b794-7ee1af0be89a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2d04c05e-6542-4f77-b794-7ee1af0be89a" (UID: "2d04c05e-6542-4f77-b794-7ee1af0be89a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:13:48.679903 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:48.679885    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d04c05e-6542-4f77-b794-7ee1af0be89a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2d04c05e-6542-4f77-b794-7ee1af0be89a" (UID: "2d04c05e-6542-4f77-b794-7ee1af0be89a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:13:48.680281 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:48.679872    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d04c05e-6542-4f77-b794-7ee1af0be89a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2d04c05e-6542-4f77-b794-7ee1af0be89a" (UID: "2d04c05e-6542-4f77-b794-7ee1af0be89a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:13:48.681756 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:48.681726    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d04c05e-6542-4f77-b794-7ee1af0be89a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2d04c05e-6542-4f77-b794-7ee1af0be89a" (UID: "2d04c05e-6542-4f77-b794-7ee1af0be89a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:13:48.681847 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:48.681800    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d04c05e-6542-4f77-b794-7ee1af0be89a-kube-api-access-vk6b9" (OuterVolumeSpecName: "kube-api-access-vk6b9") pod "2d04c05e-6542-4f77-b794-7ee1af0be89a" (UID: "2d04c05e-6542-4f77-b794-7ee1af0be89a"). InnerVolumeSpecName "kube-api-access-vk6b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:13:48.780357 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:48.780331    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk6b9\" (UniqueName: \"kubernetes.io/projected/2d04c05e-6542-4f77-b794-7ee1af0be89a-kube-api-access-vk6b9\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:13:48.780357 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:48.780357    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d04c05e-6542-4f77-b794-7ee1af0be89a-cabundle-cert\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:13:48.780524 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:48.780370    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d04c05e-6542-4f77-b794-7ee1af0be89a-kserve-provision-location\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:13:48.780524 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:48.780382    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d04c05e-6542-4f77-b794-7ee1af0be89a-proxy-tls\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:13:48.780524 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:48.780395    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d04c05e-6542-4f77-b794-7ee1af0be89a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:13:49.230288 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.230261    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm"]
Sep 17 18:13:49.230577 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.230562    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce7faf98-4168-4b83-8889-d5499f2b0a9d" containerName="storage-initializer"
Sep 17 18:13:49.230630 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.230578    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce7faf98-4168-4b83-8889-d5499f2b0a9d" containerName="storage-initializer"
Sep 17 18:13:49.230630 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.230593    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d04c05e-6542-4f77-b794-7ee1af0be89a" containerName="kube-rbac-proxy"
Sep 17 18:13:49.230630 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.230598    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d04c05e-6542-4f77-b794-7ee1af0be89a" containerName="kube-rbac-proxy"
Sep 17 18:13:49.230630 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.230611    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d04c05e-6542-4f77-b794-7ee1af0be89a" containerName="kserve-container"
Sep 17 18:13:49.230630 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.230616    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d04c05e-6542-4f77-b794-7ee1af0be89a" containerName="kserve-container"
Sep 17 18:13:49.230630 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.230625    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d04c05e-6542-4f77-b794-7ee1af0be89a" containerName="storage-initializer"
Sep 17 18:13:49.230630 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.230631    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d04c05e-6542-4f77-b794-7ee1af0be89a" containerName="storage-initializer"
Sep 17 18:13:49.230859 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.230678    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce7faf98-4168-4b83-8889-d5499f2b0a9d" containerName="storage-initializer"
Sep 17 18:13:49.230859 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.230686    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d04c05e-6542-4f77-b794-7ee1af0be89a" containerName="kserve-container"
Sep 17 18:13:49.230859 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.230691    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce7faf98-4168-4b83-8889-d5499f2b0a9d" containerName="storage-initializer"
Sep 17 18:13:49.230859 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.230697    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d04c05e-6542-4f77-b794-7ee1af0be89a" containerName="kube-rbac-proxy"
Sep 17 18:13:49.230859 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.230746    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce7faf98-4168-4b83-8889-d5499f2b0a9d" containerName="storage-initializer"
Sep 17 18:13:49.230859 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.230752    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce7faf98-4168-4b83-8889-d5499f2b0a9d" containerName="storage-initializer"
Sep 17 18:13:49.234420 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.234395    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm"
Sep 17 18:13:49.236213 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.236190    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-aee0cc-predictor-serving-cert\""
Sep 17 18:13:49.236327 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.236251    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-aee0cc-kube-rbac-proxy-sar-config\""
Sep 17 18:13:49.236515 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.236500    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-aee0cc\""
Sep 17 18:13:49.236604 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.236586    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-aee0cc-dockercfg-sqfkd\""
Sep 17 18:13:49.245386 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.245364    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm"]
Sep 17 18:13:49.385427 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.385397    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d368be0-b6c1-4f16-a299-3b9db471703d-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm\" (UID: \"6d368be0-b6c1-4f16-a299-3b9db471703d\") " pod="kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm"
Sep 17 18:13:49.385578 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.385437    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz8nr\" (UniqueName: \"kubernetes.io/projected/6d368be0-b6c1-4f16-a299-3b9db471703d-kube-api-access-tz8nr\") pod \"isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm\" (UID: \"6d368be0-b6c1-4f16-a299-3b9db471703d\") " pod="kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm"
Sep 17 18:13:49.385578 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.385481    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d368be0-b6c1-4f16-a299-3b9db471703d-proxy-tls\") pod \"isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm\" (UID: \"6d368be0-b6c1-4f16-a299-3b9db471703d\") " pod="kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm"
Sep 17 18:13:49.385578 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.385516    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d368be0-b6c1-4f16-a299-3b9db471703d-cabundle-cert\") pod \"isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm\" (UID: \"6d368be0-b6c1-4f16-a299-3b9db471703d\") " pod="kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm"
Sep 17 18:13:49.385578 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.385543    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d368be0-b6c1-4f16-a299-3b9db471703d-kserve-provision-location\") pod \"isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm\" (UID: \"6d368be0-b6c1-4f16-a299-3b9db471703d\") " pod="kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm"
Sep 17 18:13:49.393170 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.393118    2581 generic.go:356] "Generic (PLEG): container finished" podID="2d04c05e-6542-4f77-b794-7ee1af0be89a" containerID="f7832f6dcaea4b7ceb17b857bb822e18ff2f68b5429915b1b55077dd98a4abca" exitCode=0
Sep 17 18:13:49.393289 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.393211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8" event={"ID":"2d04c05e-6542-4f77-b794-7ee1af0be89a","Type":"ContainerDied","Data":"f7832f6dcaea4b7ceb17b857bb822e18ff2f68b5429915b1b55077dd98a4abca"}
Sep 17 18:13:49.393289 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.393266    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8" event={"ID":"2d04c05e-6542-4f77-b794-7ee1af0be89a","Type":"ContainerDied","Data":"9f9cf7fc493c9ea256a262e5fbe25a602e777b6362547df0a24c5d19640ba6fe"}
Sep 17 18:13:49.393378 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.393294    2581 scope.go:117] "RemoveContainer" containerID="3c7c3ddba00e5dd7bee7b2687f47ed71ae55ace00322b68ea76eacd2fa390394"
Sep 17 18:13:49.393378 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.393218    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8"
Sep 17 18:13:49.401977 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.401962    2581 scope.go:117] "RemoveContainer" containerID="f7832f6dcaea4b7ceb17b857bb822e18ff2f68b5429915b1b55077dd98a4abca"
Sep 17 18:13:49.409290 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.409274    2581 scope.go:117] "RemoveContainer" containerID="1432fbc68462c1b20a7452fb9118dc82c206434f3b535d102c577b1db126902f"
Sep 17 18:13:49.415644 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.415625    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8"]
Sep 17 18:13:49.416886 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.416874    2581 scope.go:117] "RemoveContainer" containerID="3c7c3ddba00e5dd7bee7b2687f47ed71ae55ace00322b68ea76eacd2fa390394"
Sep 17 18:13:49.417146 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:13:49.417125    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c7c3ddba00e5dd7bee7b2687f47ed71ae55ace00322b68ea76eacd2fa390394\": container with ID starting with 3c7c3ddba00e5dd7bee7b2687f47ed71ae55ace00322b68ea76eacd2fa390394 not found: ID does not exist" containerID="3c7c3ddba00e5dd7bee7b2687f47ed71ae55ace00322b68ea76eacd2fa390394"
Sep 17 18:13:49.417257 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.417173    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c7c3ddba00e5dd7bee7b2687f47ed71ae55ace00322b68ea76eacd2fa390394"} err="failed to get container status \"3c7c3ddba00e5dd7bee7b2687f47ed71ae55ace00322b68ea76eacd2fa390394\": rpc error: code = NotFound desc = could not find container \"3c7c3ddba00e5dd7bee7b2687f47ed71ae55ace00322b68ea76eacd2fa390394\": container with ID starting with 3c7c3ddba00e5dd7bee7b2687f47ed71ae55ace00322b68ea76eacd2fa390394 not found: ID does not exist"
Sep 17 18:13:49.417257 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.417197    2581 scope.go:117] "RemoveContainer" containerID="f7832f6dcaea4b7ceb17b857bb822e18ff2f68b5429915b1b55077dd98a4abca"
Sep 17 18:13:49.417462 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:13:49.417446    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7832f6dcaea4b7ceb17b857bb822e18ff2f68b5429915b1b55077dd98a4abca\": container with ID starting with f7832f6dcaea4b7ceb17b857bb822e18ff2f68b5429915b1b55077dd98a4abca not found: ID does not exist" containerID="f7832f6dcaea4b7ceb17b857bb822e18ff2f68b5429915b1b55077dd98a4abca"
Sep 17 18:13:49.417506 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.417466    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7832f6dcaea4b7ceb17b857bb822e18ff2f68b5429915b1b55077dd98a4abca"} err="failed to get container status \"f7832f6dcaea4b7ceb17b857bb822e18ff2f68b5429915b1b55077dd98a4abca\": rpc error: code = NotFound desc = could not find container \"f7832f6dcaea4b7ceb17b857bb822e18ff2f68b5429915b1b55077dd98a4abca\": container with ID starting with f7832f6dcaea4b7ceb17b857bb822e18ff2f68b5429915b1b55077dd98a4abca not found: ID does not exist"
Sep 17 18:13:49.417506 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.417480    2581 scope.go:117] "RemoveContainer" containerID="1432fbc68462c1b20a7452fb9118dc82c206434f3b535d102c577b1db126902f"
Sep 17 18:13:49.417692 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:13:49.417678    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1432fbc68462c1b20a7452fb9118dc82c206434f3b535d102c577b1db126902f\": container with ID starting with 1432fbc68462c1b20a7452fb9118dc82c206434f3b535d102c577b1db126902f not found: ID does not exist" containerID="1432fbc68462c1b20a7452fb9118dc82c206434f3b535d102c577b1db126902f"
Sep 17 18:13:49.417736 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.417697    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1432fbc68462c1b20a7452fb9118dc82c206434f3b535d102c577b1db126902f"} err="failed to get container status \"1432fbc68462c1b20a7452fb9118dc82c206434f3b535d102c577b1db126902f\": rpc error: code = NotFound desc = could not find container \"1432fbc68462c1b20a7452fb9118dc82c206434f3b535d102c577b1db126902f\": container with ID starting with 1432fbc68462c1b20a7452fb9118dc82c206434f3b535d102c577b1db126902f not found: ID does not exist"
Sep 17 18:13:49.421139 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.421119    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c81569-predictor-d8cc8857-btdk8"]
Sep 17 18:13:49.486505 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.486454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d368be0-b6c1-4f16-a299-3b9db471703d-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm\" (UID: \"6d368be0-b6c1-4f16-a299-3b9db471703d\") " pod="kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm"
Sep 17 18:13:49.486505 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.486489    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz8nr\" (UniqueName: \"kubernetes.io/projected/6d368be0-b6c1-4f16-a299-3b9db471703d-kube-api-access-tz8nr\") pod \"isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm\" (UID: \"6d368be0-b6c1-4f16-a299-3b9db471703d\") " pod="kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm"
Sep 17 18:13:49.486630 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.486516    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d368be0-b6c1-4f16-a299-3b9db471703d-proxy-tls\") pod \"isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm\" (UID: \"6d368be0-b6c1-4f16-a299-3b9db471703d\") " pod="kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm"
Sep 17 18:13:49.486630 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.486552    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d368be0-b6c1-4f16-a299-3b9db471703d-cabundle-cert\") pod \"isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm\" (UID: \"6d368be0-b6c1-4f16-a299-3b9db471703d\") " pod="kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm"
Sep 17 18:13:49.486630 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.486575    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d368be0-b6c1-4f16-a299-3b9db471703d-kserve-provision-location\") pod \"isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm\" (UID: \"6d368be0-b6c1-4f16-a299-3b9db471703d\") " pod="kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm"
Sep 17 18:13:49.486933 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.486910    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d368be0-b6c1-4f16-a299-3b9db471703d-kserve-provision-location\") pod \"isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm\" (UID: \"6d368be0-b6c1-4f16-a299-3b9db471703d\") " pod="kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm"
Sep 17 18:13:49.487208 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.487190    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d368be0-b6c1-4f16-a299-3b9db471703d-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm\" (UID: \"6d368be0-b6c1-4f16-a299-3b9db471703d\") " pod="kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm"
Sep 17 18:13:49.487264 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.487193    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d368be0-b6c1-4f16-a299-3b9db471703d-cabundle-cert\") pod \"isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm\" (UID: \"6d368be0-b6c1-4f16-a299-3b9db471703d\") " pod="kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm"
Sep 17 18:13:49.488969 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.488952    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d368be0-b6c1-4f16-a299-3b9db471703d-proxy-tls\") pod \"isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm\" (UID: \"6d368be0-b6c1-4f16-a299-3b9db471703d\") " pod="kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm"
Sep 17 18:13:49.494167 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.494112    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz8nr\" (UniqueName: \"kubernetes.io/projected/6d368be0-b6c1-4f16-a299-3b9db471703d-kube-api-access-tz8nr\") pod \"isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm\" (UID: \"6d368be0-b6c1-4f16-a299-3b9db471703d\") " pod="kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm"
Sep 17 18:13:49.544763 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.544736    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm"
Sep 17 18:13:49.819675 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:49.819501    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm"]
Sep 17 18:13:49.822203 ip-10-0-134-209 kubenswrapper[2581]: W0917 18:13:49.822174    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d368be0_b6c1_4f16_a299_3b9db471703d.slice/crio-bac716296756cde6f31c9d8e0edf87aedb8f9499bc09b328cc21e737cbe63dae WatchSource:0}: Error finding container bac716296756cde6f31c9d8e0edf87aedb8f9499bc09b328cc21e737cbe63dae: Status 404 returned error can't find the container with id bac716296756cde6f31c9d8e0edf87aedb8f9499bc09b328cc21e737cbe63dae
Sep 17 18:13:50.048423 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:50.048390    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d04c05e-6542-4f77-b794-7ee1af0be89a" path="/var/lib/kubelet/pods/2d04c05e-6542-4f77-b794-7ee1af0be89a/volumes"
Sep 17 18:13:50.404330 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:50.404297    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm" event={"ID":"6d368be0-b6c1-4f16-a299-3b9db471703d","Type":"ContainerStarted","Data":"569719290b0d73b4a56693a45983b6d56183f6b95cb16da17c096a737c73627d"}
Sep 17 18:13:50.404534 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:50.404382    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm" event={"ID":"6d368be0-b6c1-4f16-a299-3b9db471703d","Type":"ContainerStarted","Data":"bac716296756cde6f31c9d8e0edf87aedb8f9499bc09b328cc21e737cbe63dae"}
Sep 17 18:13:52.419581 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:52.419560    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm_6d368be0-b6c1-4f16-a299-3b9db471703d/storage-initializer/0.log"
Sep 17 18:13:52.419947 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:52.419597    2581 generic.go:356] "Generic (PLEG): container finished" podID="6d368be0-b6c1-4f16-a299-3b9db471703d" containerID="569719290b0d73b4a56693a45983b6d56183f6b95cb16da17c096a737c73627d" exitCode=1
Sep 17 18:13:52.419947 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:52.419687    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm" event={"ID":"6d368be0-b6c1-4f16-a299-3b9db471703d","Type":"ContainerDied","Data":"569719290b0d73b4a56693a45983b6d56183f6b95cb16da17c096a737c73627d"}
Sep 17 18:13:53.425672 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:53.425643    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm_6d368be0-b6c1-4f16-a299-3b9db471703d/storage-initializer/0.log"
Sep 17 18:13:53.426016 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:53.425761    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm" event={"ID":"6d368be0-b6c1-4f16-a299-3b9db471703d","Type":"ContainerStarted","Data":"ac21101364ba3e2f32195837b7fc64d74c6105310445172bbefffe7b2024c4a2"}
Sep 17 18:13:54.206324 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:54.206293    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm"]
Sep 17 18:13:54.430639 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:54.430598    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm" podUID="6d368be0-b6c1-4f16-a299-3b9db471703d" containerName="storage-initializer" containerID="cri-o://ac21101364ba3e2f32195837b7fc64d74c6105310445172bbefffe7b2024c4a2" gracePeriod=30
Sep 17 18:13:57.161220 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.161190    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm_6d368be0-b6c1-4f16-a299-3b9db471703d/storage-initializer/1.log"
Sep 17 18:13:57.161566 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.161539    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm_6d368be0-b6c1-4f16-a299-3b9db471703d/storage-initializer/0.log"
Sep 17 18:13:57.163890 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.163870    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm"
Sep 17 18:13:57.243185 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.243136    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d368be0-b6c1-4f16-a299-3b9db471703d-kserve-provision-location\") pod \"6d368be0-b6c1-4f16-a299-3b9db471703d\" (UID: \"6d368be0-b6c1-4f16-a299-3b9db471703d\") "
Sep 17 18:13:57.243353 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.243201    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d368be0-b6c1-4f16-a299-3b9db471703d-cabundle-cert\") pod \"6d368be0-b6c1-4f16-a299-3b9db471703d\" (UID: \"6d368be0-b6c1-4f16-a299-3b9db471703d\") "
Sep 17 18:13:57.243353 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.243254    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d368be0-b6c1-4f16-a299-3b9db471703d-kube-rbac-proxy-sar-config\") pod \"6d368be0-b6c1-4f16-a299-3b9db471703d\" (UID: \"6d368be0-b6c1-4f16-a299-3b9db471703d\") "
Sep 17 18:13:57.243353 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.243306    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tz8nr\" (UniqueName: \"kubernetes.io/projected/6d368be0-b6c1-4f16-a299-3b9db471703d-kube-api-access-tz8nr\") pod \"6d368be0-b6c1-4f16-a299-3b9db471703d\" (UID: \"6d368be0-b6c1-4f16-a299-3b9db471703d\") "
Sep 17 18:13:57.243353 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.243349    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d368be0-b6c1-4f16-a299-3b9db471703d-proxy-tls\") pod \"6d368be0-b6c1-4f16-a299-3b9db471703d\" (UID: \"6d368be0-b6c1-4f16-a299-3b9db471703d\") "
Sep 17 18:13:57.243561 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.243485    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d368be0-b6c1-4f16-a299-3b9db471703d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6d368be0-b6c1-4f16-a299-3b9db471703d" (UID: "6d368be0-b6c1-4f16-a299-3b9db471703d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:13:57.243614 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.243560    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d368be0-b6c1-4f16-a299-3b9db471703d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6d368be0-b6c1-4f16-a299-3b9db471703d" (UID: "6d368be0-b6c1-4f16-a299-3b9db471703d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:13:57.243614 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.243574    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d368be0-b6c1-4f16-a299-3b9db471703d-kserve-provision-location\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:13:57.243699 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.243669    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d368be0-b6c1-4f16-a299-3b9db471703d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6d368be0-b6c1-4f16-a299-3b9db471703d" (UID: "6d368be0-b6c1-4f16-a299-3b9db471703d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:13:57.245370 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.245339    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d368be0-b6c1-4f16-a299-3b9db471703d-kube-api-access-tz8nr" (OuterVolumeSpecName: "kube-api-access-tz8nr") pod "6d368be0-b6c1-4f16-a299-3b9db471703d" (UID: "6d368be0-b6c1-4f16-a299-3b9db471703d"). InnerVolumeSpecName "kube-api-access-tz8nr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:13:57.245506 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.245460    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d368be0-b6c1-4f16-a299-3b9db471703d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6d368be0-b6c1-4f16-a299-3b9db471703d" (UID: "6d368be0-b6c1-4f16-a299-3b9db471703d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:13:57.344628 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.344573    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d368be0-b6c1-4f16-a299-3b9db471703d-proxy-tls\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:13:57.344628 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.344594    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d368be0-b6c1-4f16-a299-3b9db471703d-cabundle-cert\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:13:57.344628 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.344604    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d368be0-b6c1-4f16-a299-3b9db471703d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:13:57.344628 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.344613    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz8nr\" (UniqueName: \"kubernetes.io/projected/6d368be0-b6c1-4f16-a299-3b9db471703d-kube-api-access-tz8nr\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:13:57.443020 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.442997    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm_6d368be0-b6c1-4f16-a299-3b9db471703d/storage-initializer/1.log"
Sep 17 18:13:57.443464 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.443450    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm_6d368be0-b6c1-4f16-a299-3b9db471703d/storage-initializer/0.log"
Sep 17 18:13:57.443527 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.443483    2581 generic.go:356] "Generic (PLEG): container finished" podID="6d368be0-b6c1-4f16-a299-3b9db471703d" containerID="ac21101364ba3e2f32195837b7fc64d74c6105310445172bbefffe7b2024c4a2" exitCode=1
Sep 17 18:13:57.443568 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.443518    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm" event={"ID":"6d368be0-b6c1-4f16-a299-3b9db471703d","Type":"ContainerDied","Data":"ac21101364ba3e2f32195837b7fc64d74c6105310445172bbefffe7b2024c4a2"}
Sep 17 18:13:57.443568 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.443544    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm"
Sep 17 18:13:57.443636 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.443580    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm" event={"ID":"6d368be0-b6c1-4f16-a299-3b9db471703d","Type":"ContainerDied","Data":"bac716296756cde6f31c9d8e0edf87aedb8f9499bc09b328cc21e737cbe63dae"}
Sep 17 18:13:57.443636 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.443608    2581 scope.go:117] "RemoveContainer" containerID="ac21101364ba3e2f32195837b7fc64d74c6105310445172bbefffe7b2024c4a2"
Sep 17 18:13:57.452109 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.452092    2581 scope.go:117] "RemoveContainer" containerID="569719290b0d73b4a56693a45983b6d56183f6b95cb16da17c096a737c73627d"
Sep 17 18:13:57.459509 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.459489    2581 scope.go:117] "RemoveContainer" containerID="ac21101364ba3e2f32195837b7fc64d74c6105310445172bbefffe7b2024c4a2"
Sep 17 18:13:57.459762 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:13:57.459744    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac21101364ba3e2f32195837b7fc64d74c6105310445172bbefffe7b2024c4a2\": container with ID starting with ac21101364ba3e2f32195837b7fc64d74c6105310445172bbefffe7b2024c4a2 not found: ID does not exist" containerID="ac21101364ba3e2f32195837b7fc64d74c6105310445172bbefffe7b2024c4a2"
Sep 17 18:13:57.459823 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.459769    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac21101364ba3e2f32195837b7fc64d74c6105310445172bbefffe7b2024c4a2"} err="failed to get container status \"ac21101364ba3e2f32195837b7fc64d74c6105310445172bbefffe7b2024c4a2\": rpc error: code = NotFound desc = could not find container \"ac21101364ba3e2f32195837b7fc64d74c6105310445172bbefffe7b2024c4a2\": container with ID starting with ac21101364ba3e2f32195837b7fc64d74c6105310445172bbefffe7b2024c4a2 not found: ID does not exist"
Sep 17 18:13:57.459823 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.459786    2581 scope.go:117] "RemoveContainer" containerID="569719290b0d73b4a56693a45983b6d56183f6b95cb16da17c096a737c73627d"
Sep 17 18:13:57.460018 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:13:57.460000    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"569719290b0d73b4a56693a45983b6d56183f6b95cb16da17c096a737c73627d\": container with ID starting with 569719290b0d73b4a56693a45983b6d56183f6b95cb16da17c096a737c73627d not found: ID does not exist" containerID="569719290b0d73b4a56693a45983b6d56183f6b95cb16da17c096a737c73627d"
Sep 17 18:13:57.460058 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.460024    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"569719290b0d73b4a56693a45983b6d56183f6b95cb16da17c096a737c73627d"} err="failed to get container status \"569719290b0d73b4a56693a45983b6d56183f6b95cb16da17c096a737c73627d\": rpc error: code = NotFound desc = could not find container \"569719290b0d73b4a56693a45983b6d56183f6b95cb16da17c096a737c73627d\": container with ID starting with 569719290b0d73b4a56693a45983b6d56183f6b95cb16da17c096a737c73627d not found: ID does not exist"
Sep 17 18:13:57.477977 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.477959    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm"]
Sep 17 18:13:57.481940 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:57.481922    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-aee0cc-predictor-8588b47b8b-sp7lm"]
Sep 17 18:13:58.049046 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:58.049015    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d368be0-b6c1-4f16-a299-3b9db471703d" path="/var/lib/kubelet/pods/6d368be0-b6c1-4f16-a299-3b9db471703d/volumes"
Sep 17 18:13:59.427140 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:59.427107    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl"]
Sep 17 18:13:59.427627 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:59.427605    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d368be0-b6c1-4f16-a299-3b9db471703d" containerName="storage-initializer"
Sep 17 18:13:59.427699 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:59.427630    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d368be0-b6c1-4f16-a299-3b9db471703d" containerName="storage-initializer"
Sep 17 18:13:59.427699 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:59.427650    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d368be0-b6c1-4f16-a299-3b9db471703d" containerName="storage-initializer"
Sep 17 18:13:59.427699 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:59.427658    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d368be0-b6c1-4f16-a299-3b9db471703d" containerName="storage-initializer"
Sep 17 18:13:59.427859 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:59.427743    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d368be0-b6c1-4f16-a299-3b9db471703d" containerName="storage-initializer"
Sep 17 18:13:59.427859 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:59.427755    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d368be0-b6c1-4f16-a299-3b9db471703d" containerName="storage-initializer"
Sep 17 18:13:59.432275 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:59.432254    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl"
Sep 17 18:13:59.434141 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:59.434119    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-e475b-kube-rbac-proxy-sar-config\""
Sep 17 18:13:59.434378 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:59.434358    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:13:59.434512 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:59.434492    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w6q8k\""
Sep 17 18:13:59.434585 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:59.434493    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:13:59.434791 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:59.434766    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:13:59.434924 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:59.434792    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:13:59.434924 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:59.434793    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-e475b-predictor-serving-cert\""
Sep 17 18:13:59.440293 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:59.440012    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl"]
Sep 17 18:13:59.461590 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:59.461559    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/779249c2-f1b9-4459-ad1a-836fe924c2b5-kserve-provision-location\") pod \"raw-sklearn-e475b-predictor-5854946c7b-f9lbl\" (UID: \"779249c2-f1b9-4459-ad1a-836fe924c2b5\") " pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl"
Sep 17 18:13:59.461736 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:59.461601    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/779249c2-f1b9-4459-ad1a-836fe924c2b5-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-e475b-predictor-5854946c7b-f9lbl\" (UID: \"779249c2-f1b9-4459-ad1a-836fe924c2b5\") " pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl"
Sep 17 18:13:59.461736 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:59.461631    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/779249c2-f1b9-4459-ad1a-836fe924c2b5-cabundle-cert\") pod \"raw-sklearn-e475b-predictor-5854946c7b-f9lbl\" (UID: \"779249c2-f1b9-4459-ad1a-836fe924c2b5\") " pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl"
Sep 17 18:13:59.461736 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:59.461699    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khp92\" (UniqueName: \"kubernetes.io/projected/779249c2-f1b9-4459-ad1a-836fe924c2b5-kube-api-access-khp92\") pod \"raw-sklearn-e475b-predictor-5854946c7b-f9lbl\" (UID: \"779249c2-f1b9-4459-ad1a-836fe924c2b5\") " pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl"
Sep 17 18:13:59.461868 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:59.461744    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/779249c2-f1b9-4459-ad1a-836fe924c2b5-proxy-tls\") pod \"raw-sklearn-e475b-predictor-5854946c7b-f9lbl\" (UID: \"779249c2-f1b9-4459-ad1a-836fe924c2b5\") " pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl"
Sep 17 18:13:59.562124 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:59.562081    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/779249c2-f1b9-4459-ad1a-836fe924c2b5-kserve-provision-location\") pod \"raw-sklearn-e475b-predictor-5854946c7b-f9lbl\" (UID: \"779249c2-f1b9-4459-ad1a-836fe924c2b5\") " pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl"
Sep 17 18:13:59.562124 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:59.562127    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/779249c2-f1b9-4459-ad1a-836fe924c2b5-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-e475b-predictor-5854946c7b-f9lbl\" (UID: \"779249c2-f1b9-4459-ad1a-836fe924c2b5\") " pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl"
Sep 17 18:13:59.562345 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:59.562177    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/779249c2-f1b9-4459-ad1a-836fe924c2b5-cabundle-cert\") pod \"raw-sklearn-e475b-predictor-5854946c7b-f9lbl\" (UID: \"779249c2-f1b9-4459-ad1a-836fe924c2b5\") " pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl"
Sep 17 18:13:59.562345 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:59.562200    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khp92\" (UniqueName: \"kubernetes.io/projected/779249c2-f1b9-4459-ad1a-836fe924c2b5-kube-api-access-khp92\") pod \"raw-sklearn-e475b-predictor-5854946c7b-f9lbl\" (UID: \"779249c2-f1b9-4459-ad1a-836fe924c2b5\") " pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl"
Sep 17 18:13:59.562345 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:59.562235    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/779249c2-f1b9-4459-ad1a-836fe924c2b5-proxy-tls\") pod \"raw-sklearn-e475b-predictor-5854946c7b-f9lbl\" (UID: \"779249c2-f1b9-4459-ad1a-836fe924c2b5\") " pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl"
Sep 17 18:13:59.562487 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:13:59.562389    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-e475b-predictor-serving-cert: secret "raw-sklearn-e475b-predictor-serving-cert" not found
Sep 17 18:13:59.562487 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:13:59.562462    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/779249c2-f1b9-4459-ad1a-836fe924c2b5-proxy-tls podName:779249c2-f1b9-4459-ad1a-836fe924c2b5 nodeName:}" failed. No retries permitted until 2026-09-17 18:14:00.062439499 +0000 UTC m=+1022.162903302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/779249c2-f1b9-4459-ad1a-836fe924c2b5-proxy-tls") pod "raw-sklearn-e475b-predictor-5854946c7b-f9lbl" (UID: "779249c2-f1b9-4459-ad1a-836fe924c2b5") : secret "raw-sklearn-e475b-predictor-serving-cert" not found
Sep 17 18:13:59.562611 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:59.562504    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/779249c2-f1b9-4459-ad1a-836fe924c2b5-kserve-provision-location\") pod \"raw-sklearn-e475b-predictor-5854946c7b-f9lbl\" (UID: \"779249c2-f1b9-4459-ad1a-836fe924c2b5\") " pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl"
Sep 17 18:13:59.562838 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:59.562818    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/779249c2-f1b9-4459-ad1a-836fe924c2b5-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-e475b-predictor-5854946c7b-f9lbl\" (UID: \"779249c2-f1b9-4459-ad1a-836fe924c2b5\") " pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl"
Sep 17 18:13:59.562922 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:59.562896    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/779249c2-f1b9-4459-ad1a-836fe924c2b5-cabundle-cert\") pod \"raw-sklearn-e475b-predictor-5854946c7b-f9lbl\" (UID: \"779249c2-f1b9-4459-ad1a-836fe924c2b5\") " pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl"
Sep 17 18:13:59.570507 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:13:59.570486    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khp92\" (UniqueName: \"kubernetes.io/projected/779249c2-f1b9-4459-ad1a-836fe924c2b5-kube-api-access-khp92\") pod \"raw-sklearn-e475b-predictor-5854946c7b-f9lbl\" (UID: \"779249c2-f1b9-4459-ad1a-836fe924c2b5\") " pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl"
Sep 17 18:14:00.067049 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:00.067023    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/779249c2-f1b9-4459-ad1a-836fe924c2b5-proxy-tls\") pod \"raw-sklearn-e475b-predictor-5854946c7b-f9lbl\" (UID: \"779249c2-f1b9-4459-ad1a-836fe924c2b5\") " pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl"
Sep 17 18:14:00.069312 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:00.069295    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/779249c2-f1b9-4459-ad1a-836fe924c2b5-proxy-tls\") pod \"raw-sklearn-e475b-predictor-5854946c7b-f9lbl\" (UID: \"779249c2-f1b9-4459-ad1a-836fe924c2b5\") " pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl"
Sep 17 18:14:00.346934 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:00.346855    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w6q8k\""
Sep 17 18:14:00.354603 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:00.354585    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl"
Sep 17 18:14:00.633245 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:00.633218    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl"]
Sep 17 18:14:00.635854 ip-10-0-134-209 kubenswrapper[2581]: W0917 18:14:00.635825    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod779249c2_f1b9_4459_ad1a_836fe924c2b5.slice/crio-5ce793d498305a0e6f3a4b2a2d18606a9d97b8a6808571960034e81b2715cf42 WatchSource:0}: Error finding container 5ce793d498305a0e6f3a4b2a2d18606a9d97b8a6808571960034e81b2715cf42: Status 404 returned error can't find the container with id 5ce793d498305a0e6f3a4b2a2d18606a9d97b8a6808571960034e81b2715cf42
Sep 17 18:14:00.639540 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:00.639521    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:14:01.465443 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:01.465411    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl" event={"ID":"779249c2-f1b9-4459-ad1a-836fe924c2b5","Type":"ContainerStarted","Data":"c22ef60dac1fbc0c8450f809ad73dac29eba0e5c2453c0eaed01c321c5c081b3"}
Sep 17 18:14:01.465626 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:01.465498    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl" event={"ID":"779249c2-f1b9-4459-ad1a-836fe924c2b5","Type":"ContainerStarted","Data":"5ce793d498305a0e6f3a4b2a2d18606a9d97b8a6808571960034e81b2715cf42"}
Sep 17 18:14:04.479417 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:04.479387    2581 generic.go:356] "Generic (PLEG): container finished" podID="779249c2-f1b9-4459-ad1a-836fe924c2b5" containerID="c22ef60dac1fbc0c8450f809ad73dac29eba0e5c2453c0eaed01c321c5c081b3" exitCode=0
Sep 17 18:14:04.479709 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:04.479460    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl" event={"ID":"779249c2-f1b9-4459-ad1a-836fe924c2b5","Type":"ContainerDied","Data":"c22ef60dac1fbc0c8450f809ad73dac29eba0e5c2453c0eaed01c321c5c081b3"}
Sep 17 18:14:05.486015 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:05.485982    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl" event={"ID":"779249c2-f1b9-4459-ad1a-836fe924c2b5","Type":"ContainerStarted","Data":"e2bc92ee1c14009ba6cc0850f0e0bbe4b9e5b1ba2f99d93e625d0137a92f12dd"}
Sep 17 18:14:05.486419 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:05.486046    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl" event={"ID":"779249c2-f1b9-4459-ad1a-836fe924c2b5","Type":"ContainerStarted","Data":"1dc389d203573a1b30971b2743dbac9e1d35df911a02f180ae869f7ef6c3493e"}
Sep 17 18:14:05.486419 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:05.486318    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl"
Sep 17 18:14:05.486547 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:05.486452    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl"
Sep 17 18:14:05.487587 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:05.487560    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl" podUID="779249c2-f1b9-4459-ad1a-836fe924c2b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 18:14:05.506831 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:05.506789    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl" podStartSLOduration=2.7896506800000003 podStartE2EDuration="6.50677523s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.71712455s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:13:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:14:05.50529566 +0000 UTC m=+1027.605759480" watchObservedRunningTime="2026-09-17 18:14:05.50677523 +0000 UTC m=+1027.607239053"
Sep 17 18:14:06.490984 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:06.490946    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl" podUID="779249c2-f1b9-4459-ad1a-836fe924c2b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 18:14:06.553821 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:06.553789    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4"]
Sep 17 18:14:06.557050 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:06.557027    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4"
Sep 17 18:14:06.558953 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:06.558936    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-91a22-kube-rbac-proxy-sar-config\""
Sep 17 18:14:06.559048 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:06.558983    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-91a22-predictor-serving-cert\""
Sep 17 18:14:06.567865 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:06.567846    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4"]
Sep 17 18:14:06.627488 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:06.627453    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbdr7\" (UniqueName: \"kubernetes.io/projected/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c-kube-api-access-lbdr7\") pod \"isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4\" (UID: \"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4"
Sep 17 18:14:06.627591 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:06.627498    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c-proxy-tls\") pod \"isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4\" (UID: \"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4"
Sep 17 18:14:06.627591 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:06.627527    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c-cabundle-cert\") pod \"isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4\" (UID: \"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4"
Sep 17 18:14:06.627681 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:06.627617    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4\" (UID: \"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4"
Sep 17 18:14:06.627681 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:06.627656    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4\" (UID: \"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4"
Sep 17 18:14:06.728993 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:06.728964    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbdr7\" (UniqueName: \"kubernetes.io/projected/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c-kube-api-access-lbdr7\") pod \"isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4\" (UID: \"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4"
Sep 17 18:14:06.729128 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:06.728997    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c-proxy-tls\") pod \"isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4\" (UID: \"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4"
Sep 17 18:14:06.729128 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:06.729019    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c-cabundle-cert\") pod \"isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4\" (UID: \"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4"
Sep 17 18:14:06.729128 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:06.729049    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4\" (UID: \"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4"
Sep 17 18:14:06.729319 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:06.729181    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4\" (UID: \"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4"
Sep 17 18:14:06.729603 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:06.729577    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4\" (UID: \"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4"
Sep 17 18:14:06.729729 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:06.729713    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c-cabundle-cert\") pod \"isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4\" (UID: \"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4"
Sep 17 18:14:06.729776 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:06.729736    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4\" (UID: \"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4"
Sep 17 18:14:06.731631 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:06.731614    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c-proxy-tls\") pod \"isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4\" (UID: \"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4"
Sep 17 18:14:06.737695 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:06.737671    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbdr7\" (UniqueName: \"kubernetes.io/projected/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c-kube-api-access-lbdr7\") pod \"isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4\" (UID: \"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4"
Sep 17 18:14:06.868031 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:06.867998    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4"
Sep 17 18:14:07.141294 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:07.141265    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4"]
Sep 17 18:14:07.143309 ip-10-0-134-209 kubenswrapper[2581]: W0917 18:14:07.143283    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea0501d_7e40_4ff8_946d_cf5bc861bf9c.slice/crio-548fb7bf0d565d542fdc91b9e23364f5f63b6bc4e096990cd0b915daa2d15ebc WatchSource:0}: Error finding container 548fb7bf0d565d542fdc91b9e23364f5f63b6bc4e096990cd0b915daa2d15ebc: Status 404 returned error can't find the container with id 548fb7bf0d565d542fdc91b9e23364f5f63b6bc4e096990cd0b915daa2d15ebc
Sep 17 18:14:07.496727 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:07.496633    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" event={"ID":"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c","Type":"ContainerStarted","Data":"213aae76e3709f63b890632c38bff6cdd6dd1418d7e28e93484c54e8d03de13e"}
Sep 17 18:14:07.496727 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:07.496703    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" event={"ID":"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c","Type":"ContainerStarted","Data":"548fb7bf0d565d542fdc91b9e23364f5f63b6bc4e096990cd0b915daa2d15ebc"}
Sep 17 18:14:11.495880 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:11.495853    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl"
Sep 17 18:14:11.496390 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:11.496361    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl" podUID="779249c2-f1b9-4459-ad1a-836fe924c2b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 18:14:11.513370 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:11.513328    2581 generic.go:356] "Generic (PLEG): container finished" podID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerID="213aae76e3709f63b890632c38bff6cdd6dd1418d7e28e93484c54e8d03de13e" exitCode=0
Sep 17 18:14:11.513501 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:11.513398    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" event={"ID":"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c","Type":"ContainerDied","Data":"213aae76e3709f63b890632c38bff6cdd6dd1418d7e28e93484c54e8d03de13e"}
Sep 17 18:14:12.520224 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:12.520188    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" event={"ID":"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c","Type":"ContainerStarted","Data":"4b6690f830d4a627cb1a8e47c00cdc45c01296ac9ad825219227c4a8e7366152"}
Sep 17 18:14:12.520603 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:12.520253    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" event={"ID":"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c","Type":"ContainerStarted","Data":"b55e6014070aa973eac1ec0e5d9826e93f3ceee14dddc7aec69b2674e3ec040b"}
Sep 17 18:14:12.520603 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:12.520591    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4"
Sep 17 18:14:12.520763 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:12.520740    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4"
Sep 17 18:14:12.522071 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:12.522047    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:14:12.538816 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:12.538752    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" podStartSLOduration=2.781032697 podStartE2EDuration="6.538736508s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.757703811s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:14:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:14:12.537285352 +0000 UTC m=+1034.637749171" watchObservedRunningTime="2026-09-17 18:14:12.538736508 +0000 UTC m=+1034.639200329"
Sep 17 18:14:13.524925 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:13.524884    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:14:14.528744 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:14.528701    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:14:19.533671 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:19.533644    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4"
Sep 17 18:14:19.534135 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:19.534097    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:14:21.496312 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:21.496272    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl" podUID="779249c2-f1b9-4459-ad1a-836fe924c2b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 18:14:29.534176 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:29.534127    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:14:31.496742 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:31.496707    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl" podUID="779249c2-f1b9-4459-ad1a-836fe924c2b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 18:14:39.534401 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:39.534362    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:14:41.497085 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:41.497008    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl" podUID="779249c2-f1b9-4459-ad1a-836fe924c2b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 18:14:49.534412 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:49.534372    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:14:51.497221 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:51.497169    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl" podUID="779249c2-f1b9-4459-ad1a-836fe924c2b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 18:14:59.535082 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:14:59.535043    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:15:01.497015 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:01.496975    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl" podUID="779249c2-f1b9-4459-ad1a-836fe924c2b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 18:15:09.534752 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:09.534713    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:15:11.497122 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:11.497074    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl" podUID="779249c2-f1b9-4459-ad1a-836fe924c2b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 18:15:19.534255 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:19.534215    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:15:21.496834 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:21.496807    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl"
Sep 17 18:15:23.044703 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:23.044664    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:15:29.538251 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:29.538218    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl"]
Sep 17 18:15:29.538626 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:29.538492    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl" podUID="779249c2-f1b9-4459-ad1a-836fe924c2b5" containerName="kserve-container" containerID="cri-o://1dc389d203573a1b30971b2743dbac9e1d35df911a02f180ae869f7ef6c3493e" gracePeriod=30
Sep 17 18:15:29.538626 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:29.538495    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl" podUID="779249c2-f1b9-4459-ad1a-836fe924c2b5" containerName="kube-rbac-proxy" containerID="cri-o://e2bc92ee1c14009ba6cc0850f0e0bbe4b9e5b1ba2f99d93e625d0137a92f12dd" gracePeriod=30
Sep 17 18:15:29.839983 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:29.839900    2581 generic.go:356] "Generic (PLEG): container finished" podID="779249c2-f1b9-4459-ad1a-836fe924c2b5" containerID="e2bc92ee1c14009ba6cc0850f0e0bbe4b9e5b1ba2f99d93e625d0137a92f12dd" exitCode=2
Sep 17 18:15:29.839983 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:29.839969    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl" event={"ID":"779249c2-f1b9-4459-ad1a-836fe924c2b5","Type":"ContainerDied","Data":"e2bc92ee1c14009ba6cc0850f0e0bbe4b9e5b1ba2f99d93e625d0137a92f12dd"}
Sep 17 18:15:31.491592 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:31.491548    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl" podUID="779249c2-f1b9-4459-ad1a-836fe924c2b5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 17 18:15:31.496396 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:31.496373    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl" podUID="779249c2-f1b9-4459-ad1a-836fe924c2b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 18:15:33.045300 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:33.045247    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:15:34.474800 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.474777    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl"
Sep 17 18:15:34.652133 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.652097    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khp92\" (UniqueName: \"kubernetes.io/projected/779249c2-f1b9-4459-ad1a-836fe924c2b5-kube-api-access-khp92\") pod \"779249c2-f1b9-4459-ad1a-836fe924c2b5\" (UID: \"779249c2-f1b9-4459-ad1a-836fe924c2b5\") "
Sep 17 18:15:34.652333 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.652140    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/779249c2-f1b9-4459-ad1a-836fe924c2b5-kube-rbac-proxy-sar-config\") pod \"779249c2-f1b9-4459-ad1a-836fe924c2b5\" (UID: \"779249c2-f1b9-4459-ad1a-836fe924c2b5\") "
Sep 17 18:15:34.652333 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.652180    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/779249c2-f1b9-4459-ad1a-836fe924c2b5-proxy-tls\") pod \"779249c2-f1b9-4459-ad1a-836fe924c2b5\" (UID: \"779249c2-f1b9-4459-ad1a-836fe924c2b5\") "
Sep 17 18:15:34.652333 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.652214    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/779249c2-f1b9-4459-ad1a-836fe924c2b5-kserve-provision-location\") pod \"779249c2-f1b9-4459-ad1a-836fe924c2b5\" (UID: \"779249c2-f1b9-4459-ad1a-836fe924c2b5\") "
Sep 17 18:15:34.652333 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.652283    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/779249c2-f1b9-4459-ad1a-836fe924c2b5-cabundle-cert\") pod \"779249c2-f1b9-4459-ad1a-836fe924c2b5\" (UID: \"779249c2-f1b9-4459-ad1a-836fe924c2b5\") "
Sep 17 18:15:34.652963 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.652838    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/779249c2-f1b9-4459-ad1a-836fe924c2b5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "779249c2-f1b9-4459-ad1a-836fe924c2b5" (UID: "779249c2-f1b9-4459-ad1a-836fe924c2b5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:15:34.652963 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.652874    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/779249c2-f1b9-4459-ad1a-836fe924c2b5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "779249c2-f1b9-4459-ad1a-836fe924c2b5" (UID: "779249c2-f1b9-4459-ad1a-836fe924c2b5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:15:34.652963 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.652960    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779249c2-f1b9-4459-ad1a-836fe924c2b5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "779249c2-f1b9-4459-ad1a-836fe924c2b5" (UID: "779249c2-f1b9-4459-ad1a-836fe924c2b5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:34.654606 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.654579    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/779249c2-f1b9-4459-ad1a-836fe924c2b5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "779249c2-f1b9-4459-ad1a-836fe924c2b5" (UID: "779249c2-f1b9-4459-ad1a-836fe924c2b5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:15:34.654748 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.654658    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/779249c2-f1b9-4459-ad1a-836fe924c2b5-kube-api-access-khp92" (OuterVolumeSpecName: "kube-api-access-khp92") pod "779249c2-f1b9-4459-ad1a-836fe924c2b5" (UID: "779249c2-f1b9-4459-ad1a-836fe924c2b5"). InnerVolumeSpecName "kube-api-access-khp92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:15:34.753981 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.753950    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/779249c2-f1b9-4459-ad1a-836fe924c2b5-cabundle-cert\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:15:34.753981 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.753978    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khp92\" (UniqueName: \"kubernetes.io/projected/779249c2-f1b9-4459-ad1a-836fe924c2b5-kube-api-access-khp92\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:15:34.753981 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.753990    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/779249c2-f1b9-4459-ad1a-836fe924c2b5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:15:34.754242 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.754000    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/779249c2-f1b9-4459-ad1a-836fe924c2b5-proxy-tls\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:15:34.754242 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.754008    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/779249c2-f1b9-4459-ad1a-836fe924c2b5-kserve-provision-location\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:15:34.862279 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.862244    2581 generic.go:356] "Generic (PLEG): container finished" podID="779249c2-f1b9-4459-ad1a-836fe924c2b5" containerID="1dc389d203573a1b30971b2743dbac9e1d35df911a02f180ae869f7ef6c3493e" exitCode=0
Sep 17 18:15:34.862445 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.862308    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl"
Sep 17 18:15:34.862445 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.862331    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl" event={"ID":"779249c2-f1b9-4459-ad1a-836fe924c2b5","Type":"ContainerDied","Data":"1dc389d203573a1b30971b2743dbac9e1d35df911a02f180ae869f7ef6c3493e"}
Sep 17 18:15:34.862445 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.862407    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl" event={"ID":"779249c2-f1b9-4459-ad1a-836fe924c2b5","Type":"ContainerDied","Data":"5ce793d498305a0e6f3a4b2a2d18606a9d97b8a6808571960034e81b2715cf42"}
Sep 17 18:15:34.862445 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.862437    2581 scope.go:117] "RemoveContainer" containerID="e2bc92ee1c14009ba6cc0850f0e0bbe4b9e5b1ba2f99d93e625d0137a92f12dd"
Sep 17 18:15:34.872144 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.872126    2581 scope.go:117] "RemoveContainer" containerID="1dc389d203573a1b30971b2743dbac9e1d35df911a02f180ae869f7ef6c3493e"
Sep 17 18:15:34.879605 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.879591    2581 scope.go:117] "RemoveContainer" containerID="c22ef60dac1fbc0c8450f809ad73dac29eba0e5c2453c0eaed01c321c5c081b3"
Sep 17 18:15:34.884304 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.884282    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl"]
Sep 17 18:15:34.887085 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.887072    2581 scope.go:117] "RemoveContainer" containerID="e2bc92ee1c14009ba6cc0850f0e0bbe4b9e5b1ba2f99d93e625d0137a92f12dd"
Sep 17 18:15:34.887402 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:15:34.887385    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2bc92ee1c14009ba6cc0850f0e0bbe4b9e5b1ba2f99d93e625d0137a92f12dd\": container with ID starting with e2bc92ee1c14009ba6cc0850f0e0bbe4b9e5b1ba2f99d93e625d0137a92f12dd not found: ID does not exist" containerID="e2bc92ee1c14009ba6cc0850f0e0bbe4b9e5b1ba2f99d93e625d0137a92f12dd"
Sep 17 18:15:34.887465 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.887414    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2bc92ee1c14009ba6cc0850f0e0bbe4b9e5b1ba2f99d93e625d0137a92f12dd"} err="failed to get container status \"e2bc92ee1c14009ba6cc0850f0e0bbe4b9e5b1ba2f99d93e625d0137a92f12dd\": rpc error: code = NotFound desc = could not find container \"e2bc92ee1c14009ba6cc0850f0e0bbe4b9e5b1ba2f99d93e625d0137a92f12dd\": container with ID starting with e2bc92ee1c14009ba6cc0850f0e0bbe4b9e5b1ba2f99d93e625d0137a92f12dd not found: ID does not exist"
Sep 17 18:15:34.887465 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.887431    2581 scope.go:117] "RemoveContainer" containerID="1dc389d203573a1b30971b2743dbac9e1d35df911a02f180ae869f7ef6c3493e"
Sep 17 18:15:34.887709 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:15:34.887691    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dc389d203573a1b30971b2743dbac9e1d35df911a02f180ae869f7ef6c3493e\": container with ID starting with 1dc389d203573a1b30971b2743dbac9e1d35df911a02f180ae869f7ef6c3493e not found: ID does not exist" containerID="1dc389d203573a1b30971b2743dbac9e1d35df911a02f180ae869f7ef6c3493e"
Sep 17 18:15:34.887822 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.887710    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dc389d203573a1b30971b2743dbac9e1d35df911a02f180ae869f7ef6c3493e"} err="failed to get container status \"1dc389d203573a1b30971b2743dbac9e1d35df911a02f180ae869f7ef6c3493e\": rpc error: code = NotFound desc = could not find container \"1dc389d203573a1b30971b2743dbac9e1d35df911a02f180ae869f7ef6c3493e\": container with ID starting with 1dc389d203573a1b30971b2743dbac9e1d35df911a02f180ae869f7ef6c3493e not found: ID does not exist"
Sep 17 18:15:34.887822 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.887723    2581 scope.go:117] "RemoveContainer" containerID="c22ef60dac1fbc0c8450f809ad73dac29eba0e5c2453c0eaed01c321c5c081b3"
Sep 17 18:15:34.888001 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:15:34.887972    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c22ef60dac1fbc0c8450f809ad73dac29eba0e5c2453c0eaed01c321c5c081b3\": container with ID starting with c22ef60dac1fbc0c8450f809ad73dac29eba0e5c2453c0eaed01c321c5c081b3 not found: ID does not exist" containerID="c22ef60dac1fbc0c8450f809ad73dac29eba0e5c2453c0eaed01c321c5c081b3"
Sep 17 18:15:34.888076 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.888001    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c22ef60dac1fbc0c8450f809ad73dac29eba0e5c2453c0eaed01c321c5c081b3"} err="failed to get container status \"c22ef60dac1fbc0c8450f809ad73dac29eba0e5c2453c0eaed01c321c5c081b3\": rpc error: code = NotFound desc = could not find container \"c22ef60dac1fbc0c8450f809ad73dac29eba0e5c2453c0eaed01c321c5c081b3\": container with ID starting with c22ef60dac1fbc0c8450f809ad73dac29eba0e5c2453c0eaed01c321c5c081b3 not found: ID does not exist"
Sep 17 18:15:34.889939 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:34.889917    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-e475b-predictor-5854946c7b-f9lbl"]
Sep 17 18:15:36.049014 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:36.048978    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="779249c2-f1b9-4459-ad1a-836fe924c2b5" path="/var/lib/kubelet/pods/779249c2-f1b9-4459-ad1a-836fe924c2b5/volumes"
Sep 17 18:15:43.045289 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:43.045252    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:15:53.045419 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:15:53.045382    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:16:03.045103 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:16:03.045061    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:16:13.044681 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:16:13.044593    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:16:23.045306 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:16:23.045261    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:16:33.045660 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:16:33.045620    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:16:38.044757 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:16:38.044717    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:16:48.048687 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:16:48.048659    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4"
Sep 17 18:16:56.621615 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:16:56.621582    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4"]
Sep 17 18:16:56.622078 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:16:56.621891    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="kserve-container" containerID="cri-o://b55e6014070aa973eac1ec0e5d9826e93f3ceee14dddc7aec69b2674e3ec040b" gracePeriod=30
Sep 17 18:16:56.622078 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:16:56.621959    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="kube-rbac-proxy" containerID="cri-o://4b6690f830d4a627cb1a8e47c00cdc45c01296ac9ad825219227c4a8e7366152" gracePeriod=30
Sep 17 18:16:57.187861 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:16:57.187830    2581 generic.go:356] "Generic (PLEG): container finished" podID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerID="4b6690f830d4a627cb1a8e47c00cdc45c01296ac9ad825219227c4a8e7366152" exitCode=2
Sep 17 18:16:57.188028 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:16:57.187888    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" event={"ID":"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c","Type":"ContainerDied","Data":"4b6690f830d4a627cb1a8e47c00cdc45c01296ac9ad825219227c4a8e7366152"}
Sep 17 18:16:58.044970 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:16:58.044929    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:16:59.529887 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:16:59.529846    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 17 18:16:59.991327 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:16:59.991296    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm8hx_913ce1e9-9fb6-4049-a06d-9ebc69d3e528/ovn-acl-logging/0.log"
Sep 17 18:16:59.993087 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:16:59.993066    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm8hx_913ce1e9-9fb6-4049-a06d-9ebc69d3e528/ovn-acl-logging/0.log"
Sep 17 18:17:01.821026 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:01.820987    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf"]
Sep 17 18:17:01.821490 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:01.821473    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="779249c2-f1b9-4459-ad1a-836fe924c2b5" containerName="kserve-container"
Sep 17 18:17:01.821578 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:01.821493    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="779249c2-f1b9-4459-ad1a-836fe924c2b5" containerName="kserve-container"
Sep 17 18:17:01.821578 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:01.821505    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="779249c2-f1b9-4459-ad1a-836fe924c2b5" containerName="kube-rbac-proxy"
Sep 17 18:17:01.821578 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:01.821513    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="779249c2-f1b9-4459-ad1a-836fe924c2b5" containerName="kube-rbac-proxy"
Sep 17 18:17:01.821578 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:01.821534    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="779249c2-f1b9-4459-ad1a-836fe924c2b5" containerName="storage-initializer"
Sep 17 18:17:01.821578 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:01.821545    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="779249c2-f1b9-4459-ad1a-836fe924c2b5" containerName="storage-initializer"
Sep 17 18:17:01.821731 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:01.821619    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="779249c2-f1b9-4459-ad1a-836fe924c2b5" containerName="kube-rbac-proxy"
Sep 17 18:17:01.821731 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:01.821630    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="779249c2-f1b9-4459-ad1a-836fe924c2b5" containerName="kserve-container"
Sep 17 18:17:01.825008 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:01.824989    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf"
Sep 17 18:17:01.826850 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:01.826824    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 17 18:17:01.826850 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:01.826845    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 18:17:01.832882 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:01.832861    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf"]
Sep 17 18:17:01.896537 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:01.896513    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk"]
Sep 17 18:17:01.899839 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:01.899822    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk"
Sep 17 18:17:01.901672 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:01.901654    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 18:17:01.909369 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:01.909346    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk"]
Sep 17 18:17:01.946350 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:01.946321    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55e57a98-c38f-4ea2-b8e8-25d876d21775-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5956b77f85-kjzjk\" (UID: \"55e57a98-c38f-4ea2-b8e8-25d876d21775\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk"
Sep 17 18:17:01.946350 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:01.946350    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t962c\" (UniqueName: \"kubernetes.io/projected/55e57a98-c38f-4ea2-b8e8-25d876d21775-kube-api-access-t962c\") pod \"isvc-canary-create-predictor-5956b77f85-kjzjk\" (UID: \"55e57a98-c38f-4ea2-b8e8-25d876d21775\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk"
Sep 17 18:17:01.946524 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:01.946389    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ea4957c-61b3-4888-a197-db7ddefbd6ea-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5bb585469b-kqcgf\" (UID: \"5ea4957c-61b3-4888-a197-db7ddefbd6ea\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf"
Sep 17 18:17:01.946524 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:01.946431    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7kdq\" (UniqueName: \"kubernetes.io/projected/5ea4957c-61b3-4888-a197-db7ddefbd6ea-kube-api-access-s7kdq\") pod \"isvc-canary-create-v2-predictor-5bb585469b-kqcgf\" (UID: \"5ea4957c-61b3-4888-a197-db7ddefbd6ea\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf"
Sep 17 18:17:01.946524 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:01.946449    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55e57a98-c38f-4ea2-b8e8-25d876d21775-cabundle-cert\") pod \"isvc-canary-create-predictor-5956b77f85-kjzjk\" (UID: \"55e57a98-c38f-4ea2-b8e8-25d876d21775\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk"
Sep 17 18:17:01.946682 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:01.946536    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ea4957c-61b3-4888-a197-db7ddefbd6ea-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5bb585469b-kqcgf\" (UID: \"5ea4957c-61b3-4888-a197-db7ddefbd6ea\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf"
Sep 17 18:17:01.946682 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:01.946594    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ea4957c-61b3-4888-a197-db7ddefbd6ea-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5bb585469b-kqcgf\" (UID: \"5ea4957c-61b3-4888-a197-db7ddefbd6ea\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf"
Sep 17 18:17:01.946682 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:01.946623    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ea4957c-61b3-4888-a197-db7ddefbd6ea-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5bb585469b-kqcgf\" (UID: \"5ea4957c-61b3-4888-a197-db7ddefbd6ea\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf"
Sep 17 18:17:01.946682 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:01.946673    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55e57a98-c38f-4ea2-b8e8-25d876d21775-proxy-tls\") pod \"isvc-canary-create-predictor-5956b77f85-kjzjk\" (UID: \"55e57a98-c38f-4ea2-b8e8-25d876d21775\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk"
Sep 17 18:17:01.946888 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:01.946690    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55e57a98-c38f-4ea2-b8e8-25d876d21775-kserve-provision-location\") pod \"isvc-canary-create-predictor-5956b77f85-kjzjk\" (UID: \"55e57a98-c38f-4ea2-b8e8-25d876d21775\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk"
Sep 17 18:17:02.047614 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:02.047586    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ea4957c-61b3-4888-a197-db7ddefbd6ea-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5bb585469b-kqcgf\" (UID: \"5ea4957c-61b3-4888-a197-db7ddefbd6ea\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf"
Sep 17 18:17:02.047614 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:02.047617    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7kdq\" (UniqueName: \"kubernetes.io/projected/5ea4957c-61b3-4888-a197-db7ddefbd6ea-kube-api-access-s7kdq\") pod \"isvc-canary-create-v2-predictor-5bb585469b-kqcgf\" (UID: \"5ea4957c-61b3-4888-a197-db7ddefbd6ea\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf"
Sep 17 18:17:02.047824 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:02.047639    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55e57a98-c38f-4ea2-b8e8-25d876d21775-cabundle-cert\") pod \"isvc-canary-create-predictor-5956b77f85-kjzjk\" (UID: \"55e57a98-c38f-4ea2-b8e8-25d876d21775\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk"
Sep 17 18:17:02.047824 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:02.047683    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ea4957c-61b3-4888-a197-db7ddefbd6ea-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5bb585469b-kqcgf\" (UID: \"5ea4957c-61b3-4888-a197-db7ddefbd6ea\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf"
Sep 17 18:17:02.047824 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:02.047714    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ea4957c-61b3-4888-a197-db7ddefbd6ea-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5bb585469b-kqcgf\" (UID: \"5ea4957c-61b3-4888-a197-db7ddefbd6ea\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf"
Sep 17 18:17:02.047824 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:17:02.047739    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 17 18:17:02.047824 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:17:02.047796    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ea4957c-61b3-4888-a197-db7ddefbd6ea-proxy-tls podName:5ea4957c-61b3-4888-a197-db7ddefbd6ea nodeName:}" failed. No retries permitted until 2026-09-17 18:17:02.547778257 +0000 UTC m=+1204.648242071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5ea4957c-61b3-4888-a197-db7ddefbd6ea-proxy-tls") pod "isvc-canary-create-v2-predictor-5bb585469b-kqcgf" (UID: "5ea4957c-61b3-4888-a197-db7ddefbd6ea") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 17 18:17:02.048093 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:02.047739    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ea4957c-61b3-4888-a197-db7ddefbd6ea-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5bb585469b-kqcgf\" (UID: \"5ea4957c-61b3-4888-a197-db7ddefbd6ea\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf"
Sep 17 18:17:02.048093 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:02.047908    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55e57a98-c38f-4ea2-b8e8-25d876d21775-proxy-tls\") pod \"isvc-canary-create-predictor-5956b77f85-kjzjk\" (UID: \"55e57a98-c38f-4ea2-b8e8-25d876d21775\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk"
Sep 17 18:17:02.048093 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:02.047971    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55e57a98-c38f-4ea2-b8e8-25d876d21775-kserve-provision-location\") pod \"isvc-canary-create-predictor-5956b77f85-kjzjk\" (UID: \"55e57a98-c38f-4ea2-b8e8-25d876d21775\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk"
Sep 17 18:17:02.048093 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:02.048010    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55e57a98-c38f-4ea2-b8e8-25d876d21775-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5956b77f85-kjzjk\" (UID: \"55e57a98-c38f-4ea2-b8e8-25d876d21775\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk"
Sep 17 18:17:02.048093 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:02.048051    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t962c\" (UniqueName: \"kubernetes.io/projected/55e57a98-c38f-4ea2-b8e8-25d876d21775-kube-api-access-t962c\") pod \"isvc-canary-create-predictor-5956b77f85-kjzjk\" (UID: \"55e57a98-c38f-4ea2-b8e8-25d876d21775\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk"
Sep 17 18:17:02.048389 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:02.048302    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ea4957c-61b3-4888-a197-db7ddefbd6ea-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5bb585469b-kqcgf\" (UID: \"5ea4957c-61b3-4888-a197-db7ddefbd6ea\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf"
Sep 17 18:17:02.048506 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:02.048474    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ea4957c-61b3-4888-a197-db7ddefbd6ea-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5bb585469b-kqcgf\" (UID: \"5ea4957c-61b3-4888-a197-db7ddefbd6ea\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf"
Sep 17 18:17:02.048657 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:02.048501    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55e57a98-c38f-4ea2-b8e8-25d876d21775-cabundle-cert\") pod \"isvc-canary-create-predictor-5956b77f85-kjzjk\" (UID: \"55e57a98-c38f-4ea2-b8e8-25d876d21775\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk"
Sep 17 18:17:02.048657 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:02.048588    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55e57a98-c38f-4ea2-b8e8-25d876d21775-kserve-provision-location\") pod \"isvc-canary-create-predictor-5956b77f85-kjzjk\" (UID: \"55e57a98-c38f-4ea2-b8e8-25d876d21775\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk"
Sep 17 18:17:02.048657 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:02.048627    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ea4957c-61b3-4888-a197-db7ddefbd6ea-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5bb585469b-kqcgf\" (UID: \"5ea4957c-61b3-4888-a197-db7ddefbd6ea\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf"
Sep 17 18:17:02.049004 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:02.048974    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55e57a98-c38f-4ea2-b8e8-25d876d21775-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5956b77f85-kjzjk\" (UID: \"55e57a98-c38f-4ea2-b8e8-25d876d21775\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk"
Sep 17 18:17:02.051029 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:02.051008    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55e57a98-c38f-4ea2-b8e8-25d876d21775-proxy-tls\") pod \"isvc-canary-create-predictor-5956b77f85-kjzjk\" (UID: \"55e57a98-c38f-4ea2-b8e8-25d876d21775\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk"
Sep 17 18:17:02.068915 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:02.068895    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7kdq\" (UniqueName: \"kubernetes.io/projected/5ea4957c-61b3-4888-a197-db7ddefbd6ea-kube-api-access-s7kdq\") pod \"isvc-canary-create-v2-predictor-5bb585469b-kqcgf\" (UID: \"5ea4957c-61b3-4888-a197-db7ddefbd6ea\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf"
Sep 17 18:17:02.069012 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:02.068983    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t962c\" (UniqueName: \"kubernetes.io/projected/55e57a98-c38f-4ea2-b8e8-25d876d21775-kube-api-access-t962c\") pod \"isvc-canary-create-predictor-5956b77f85-kjzjk\" (UID: \"55e57a98-c38f-4ea2-b8e8-25d876d21775\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk"
Sep 17 18:17:02.210447 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:02.210414    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk"
Sep 17 18:17:02.482332 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:02.482245    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk"]
Sep 17 18:17:02.486554 ip-10-0-134-209 kubenswrapper[2581]: W0917 18:17:02.486530    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e57a98_c38f_4ea2_b8e8_25d876d21775.slice/crio-43b1f9cd96eac30e4e713c9aee6cbd5a19147ffc3b41f974c436094b93c770cc WatchSource:0}: Error finding container 43b1f9cd96eac30e4e713c9aee6cbd5a19147ffc3b41f974c436094b93c770cc: Status 404 returned error can't find the container with id 43b1f9cd96eac30e4e713c9aee6cbd5a19147ffc3b41f974c436094b93c770cc
Sep 17 18:17:02.552997 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:02.552972    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ea4957c-61b3-4888-a197-db7ddefbd6ea-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5bb585469b-kqcgf\" (UID: \"5ea4957c-61b3-4888-a197-db7ddefbd6ea\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf"
Sep 17 18:17:02.555339 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:02.555320    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ea4957c-61b3-4888-a197-db7ddefbd6ea-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5bb585469b-kqcgf\" (UID: \"5ea4957c-61b3-4888-a197-db7ddefbd6ea\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf"
Sep 17 18:17:02.736554 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:02.736471    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf"
Sep 17 18:17:03.011877 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:03.011783    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf"]
Sep 17 18:17:03.015224 ip-10-0-134-209 kubenswrapper[2581]: W0917 18:17:03.015201    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea4957c_61b3_4888_a197_db7ddefbd6ea.slice/crio-d0c50f5a7e34a4a0c46e3fa63314114d737ad57d3b92298eb3e7708ccce91969 WatchSource:0}: Error finding container d0c50f5a7e34a4a0c46e3fa63314114d737ad57d3b92298eb3e7708ccce91969: Status 404 returned error can't find the container with id d0c50f5a7e34a4a0c46e3fa63314114d737ad57d3b92298eb3e7708ccce91969
Sep 17 18:17:03.211509 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:03.211454    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf" event={"ID":"5ea4957c-61b3-4888-a197-db7ddefbd6ea","Type":"ContainerStarted","Data":"208784c947986ae567a826292a93d0b2ac8fc4b53f49a44037f7b19971f92c66"}
Sep 17 18:17:03.211708 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:03.211541    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf" event={"ID":"5ea4957c-61b3-4888-a197-db7ddefbd6ea","Type":"ContainerStarted","Data":"d0c50f5a7e34a4a0c46e3fa63314114d737ad57d3b92298eb3e7708ccce91969"}
Sep 17 18:17:03.213210 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:03.213183    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk" event={"ID":"55e57a98-c38f-4ea2-b8e8-25d876d21775","Type":"ContainerStarted","Data":"0f298e368cc8c3ac27625c38092dbb28223bb45fc5dc52c9691a807f524b6028"}
Sep 17 18:17:03.213327 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:03.213244    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk" event={"ID":"55e57a98-c38f-4ea2-b8e8-25d876d21775","Type":"ContainerStarted","Data":"43b1f9cd96eac30e4e713c9aee6cbd5a19147ffc3b41f974c436094b93c770cc"}
Sep 17 18:17:04.529080 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:04.529046    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 17 18:17:06.064454 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.064433    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4"
Sep 17 18:17:06.183030 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.182950    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbdr7\" (UniqueName: \"kubernetes.io/projected/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c-kube-api-access-lbdr7\") pod \"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c\" (UID: \"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c\") "
Sep 17 18:17:06.183030 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.182991    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c-kserve-provision-location\") pod \"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c\" (UID: \"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c\") "
Sep 17 18:17:06.183030 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.183011    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c-cabundle-cert\") pod \"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c\" (UID: \"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c\") "
Sep 17 18:17:06.183323 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.183048    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c-kube-rbac-proxy-sar-config\") pod \"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c\" (UID: \"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c\") "
Sep 17 18:17:06.183323 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.183080    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c-proxy-tls\") pod \"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c\" (UID: \"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c\") "
Sep 17 18:17:06.183417 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.183308    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" (UID: "3ea0501d-7e40-4ff8-946d-cf5bc861bf9c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:17:06.183462 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.183420    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" (UID: "3ea0501d-7e40-4ff8-946d-cf5bc861bf9c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:17:06.183507 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.183455    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" (UID: "3ea0501d-7e40-4ff8-946d-cf5bc861bf9c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:17:06.185426 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.185406    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" (UID: "3ea0501d-7e40-4ff8-946d-cf5bc861bf9c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:17:06.185659 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.185642    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c-kube-api-access-lbdr7" (OuterVolumeSpecName: "kube-api-access-lbdr7") pod "3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" (UID: "3ea0501d-7e40-4ff8-946d-cf5bc861bf9c"). InnerVolumeSpecName "kube-api-access-lbdr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:17:06.226996 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.226973    2581 generic.go:356] "Generic (PLEG): container finished" podID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerID="b55e6014070aa973eac1ec0e5d9826e93f3ceee14dddc7aec69b2674e3ec040b" exitCode=0
Sep 17 18:17:06.227093 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.227004    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" event={"ID":"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c","Type":"ContainerDied","Data":"b55e6014070aa973eac1ec0e5d9826e93f3ceee14dddc7aec69b2674e3ec040b"}
Sep 17 18:17:06.227093 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.227031    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4"
Sep 17 18:17:06.227093 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.227050    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4" event={"ID":"3ea0501d-7e40-4ff8-946d-cf5bc861bf9c","Type":"ContainerDied","Data":"548fb7bf0d565d542fdc91b9e23364f5f63b6bc4e096990cd0b915daa2d15ebc"}
Sep 17 18:17:06.227093 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.227076    2581 scope.go:117] "RemoveContainer" containerID="4b6690f830d4a627cb1a8e47c00cdc45c01296ac9ad825219227c4a8e7366152"
Sep 17 18:17:06.234996 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.234980    2581 scope.go:117] "RemoveContainer" containerID="b55e6014070aa973eac1ec0e5d9826e93f3ceee14dddc7aec69b2674e3ec040b"
Sep 17 18:17:06.241917 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.241900    2581 scope.go:117] "RemoveContainer" containerID="213aae76e3709f63b890632c38bff6cdd6dd1418d7e28e93484c54e8d03de13e"
Sep 17 18:17:06.248644 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.248621    2581 scope.go:117] "RemoveContainer" containerID="4b6690f830d4a627cb1a8e47c00cdc45c01296ac9ad825219227c4a8e7366152"
Sep 17 18:17:06.248893 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:17:06.248875    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b6690f830d4a627cb1a8e47c00cdc45c01296ac9ad825219227c4a8e7366152\": container with ID starting with 4b6690f830d4a627cb1a8e47c00cdc45c01296ac9ad825219227c4a8e7366152 not found: ID does not exist" containerID="4b6690f830d4a627cb1a8e47c00cdc45c01296ac9ad825219227c4a8e7366152"
Sep 17 18:17:06.248952 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.248899    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b6690f830d4a627cb1a8e47c00cdc45c01296ac9ad825219227c4a8e7366152"} err="failed to get container status \"4b6690f830d4a627cb1a8e47c00cdc45c01296ac9ad825219227c4a8e7366152\": rpc error: code = NotFound desc = could not find container \"4b6690f830d4a627cb1a8e47c00cdc45c01296ac9ad825219227c4a8e7366152\": container with ID starting with 4b6690f830d4a627cb1a8e47c00cdc45c01296ac9ad825219227c4a8e7366152 not found: ID does not exist"
Sep 17 18:17:06.248952 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.248915    2581 scope.go:117] "RemoveContainer" containerID="b55e6014070aa973eac1ec0e5d9826e93f3ceee14dddc7aec69b2674e3ec040b"
Sep 17 18:17:06.249182 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:17:06.249135    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b55e6014070aa973eac1ec0e5d9826e93f3ceee14dddc7aec69b2674e3ec040b\": container with ID starting with b55e6014070aa973eac1ec0e5d9826e93f3ceee14dddc7aec69b2674e3ec040b not found: ID does not exist" containerID="b55e6014070aa973eac1ec0e5d9826e93f3ceee14dddc7aec69b2674e3ec040b"
Sep 17 18:17:06.249230 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.249188    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b55e6014070aa973eac1ec0e5d9826e93f3ceee14dddc7aec69b2674e3ec040b"} err="failed to get container status \"b55e6014070aa973eac1ec0e5d9826e93f3ceee14dddc7aec69b2674e3ec040b\": rpc error: code = NotFound desc = could not find container \"b55e6014070aa973eac1ec0e5d9826e93f3ceee14dddc7aec69b2674e3ec040b\": container with ID starting with b55e6014070aa973eac1ec0e5d9826e93f3ceee14dddc7aec69b2674e3ec040b not found: ID does not exist"
Sep 17 18:17:06.249230 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.249201    2581 scope.go:117] "RemoveContainer" containerID="213aae76e3709f63b890632c38bff6cdd6dd1418d7e28e93484c54e8d03de13e"
Sep 17 18:17:06.249426 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:17:06.249410    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"213aae76e3709f63b890632c38bff6cdd6dd1418d7e28e93484c54e8d03de13e\": container with ID starting with 213aae76e3709f63b890632c38bff6cdd6dd1418d7e28e93484c54e8d03de13e not found: ID does not exist" containerID="213aae76e3709f63b890632c38bff6cdd6dd1418d7e28e93484c54e8d03de13e"
Sep 17 18:17:06.249469 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.249429    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"213aae76e3709f63b890632c38bff6cdd6dd1418d7e28e93484c54e8d03de13e"} err="failed to get container status \"213aae76e3709f63b890632c38bff6cdd6dd1418d7e28e93484c54e8d03de13e\": rpc error: code = NotFound desc = could not find container \"213aae76e3709f63b890632c38bff6cdd6dd1418d7e28e93484c54e8d03de13e\": container with ID starting with 213aae76e3709f63b890632c38bff6cdd6dd1418d7e28e93484c54e8d03de13e not found: ID does not exist"
Sep 17 18:17:06.257981 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.257956    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4"]
Sep 17 18:17:06.266488 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.266469    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-91a22-predictor-85679b8fcd-kqzp4"]
Sep 17 18:17:06.284286 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.284267    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbdr7\" (UniqueName: \"kubernetes.io/projected/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c-kube-api-access-lbdr7\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:17:06.284346 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.284289    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c-kserve-provision-location\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:17:06.284346 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.284300    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c-cabundle-cert\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:17:06.284346 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.284308    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:17:06.284346 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:06.284316    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c-proxy-tls\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:17:07.232312 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:07.232233    2581 generic.go:356] "Generic (PLEG): container finished" podID="55e57a98-c38f-4ea2-b8e8-25d876d21775" containerID="0f298e368cc8c3ac27625c38092dbb28223bb45fc5dc52c9691a807f524b6028" exitCode=0
Sep 17 18:17:07.232312 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:07.232297    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk" event={"ID":"55e57a98-c38f-4ea2-b8e8-25d876d21775","Type":"ContainerDied","Data":"0f298e368cc8c3ac27625c38092dbb28223bb45fc5dc52c9691a807f524b6028"}
Sep 17 18:17:07.234712 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:07.234692    2581 generic.go:356] "Generic (PLEG): container finished" podID="5ea4957c-61b3-4888-a197-db7ddefbd6ea" containerID="208784c947986ae567a826292a93d0b2ac8fc4b53f49a44037f7b19971f92c66" exitCode=0
Sep 17 18:17:07.234810 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:07.234730    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf" event={"ID":"5ea4957c-61b3-4888-a197-db7ddefbd6ea","Type":"ContainerDied","Data":"208784c947986ae567a826292a93d0b2ac8fc4b53f49a44037f7b19971f92c66"}
Sep 17 18:17:08.048745 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:08.048711    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" path="/var/lib/kubelet/pods/3ea0501d-7e40-4ff8-946d-cf5bc861bf9c/volumes"
Sep 17 18:17:08.240228 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:08.240186    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk" event={"ID":"55e57a98-c38f-4ea2-b8e8-25d876d21775","Type":"ContainerStarted","Data":"24f7c37c945da1198a971e7f0f2d72a8fb0bfeca20b2994ef828f9bc744526eb"}
Sep 17 18:17:08.240634 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:08.240250    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk" event={"ID":"55e57a98-c38f-4ea2-b8e8-25d876d21775","Type":"ContainerStarted","Data":"a44022d4dffdb44f17df05243d980bce89f9afb8c115be82d245a66c47a3ca2c"}
Sep 17 18:17:08.240634 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:08.240539    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk"
Sep 17 18:17:08.240733 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:08.240689    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk"
Sep 17 18:17:08.242144 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:08.242119    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk" podUID="55e57a98-c38f-4ea2-b8e8-25d876d21775" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:17:08.242387 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:08.242369    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf" event={"ID":"5ea4957c-61b3-4888-a197-db7ddefbd6ea","Type":"ContainerStarted","Data":"11f402ccfd5b099ef00560806ee6d1ee5d5ab5c522c2e05eeed1e620f71ff4c8"}
Sep 17 18:17:08.242431 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:08.242410    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf" event={"ID":"5ea4957c-61b3-4888-a197-db7ddefbd6ea","Type":"ContainerStarted","Data":"e9a245cbaeb21aaf1f6979902e5ec01a4d601d79f2a1c351fcb621f61ca20571"}
Sep 17 18:17:08.242594 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:08.242580    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf"
Sep 17 18:17:08.257392 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:08.257343    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk" podStartSLOduration=3.217567121 podStartE2EDuration="7.257326052s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.039758931s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:17:08.256529191 +0000 UTC m=+1210.356993009" watchObservedRunningTime="2026-09-17 18:17:08.257326052 +0000 UTC m=+1210.357789872"
Sep 17 18:17:08.274030 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:08.273989    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf" podStartSLOduration=3.366809876 podStartE2EDuration="7.273974469s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.907164593s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:17:08.27302614 +0000 UTC m=+1210.373489958" watchObservedRunningTime="2026-09-17 18:17:08.273974469 +0000 UTC m=+1210.374438290"
Sep 17 18:17:09.246616 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:09.246579    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk" podUID="55e57a98-c38f-4ea2-b8e8-25d876d21775" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:17:09.247055 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:09.246712    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf"
Sep 17 18:17:09.248047 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:09.248021    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf" podUID="5ea4957c-61b3-4888-a197-db7ddefbd6ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:17:10.253535 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:10.253497    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf" podUID="5ea4957c-61b3-4888-a197-db7ddefbd6ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:17:14.250863 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:14.250834    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk"
Sep 17 18:17:14.251398 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:14.251370    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk" podUID="55e57a98-c38f-4ea2-b8e8-25d876d21775" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:17:15.259226 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:15.259200    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf"
Sep 17 18:17:15.259727 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:15.259699    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf" podUID="5ea4957c-61b3-4888-a197-db7ddefbd6ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:17:24.251553 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:24.251512    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk" podUID="55e57a98-c38f-4ea2-b8e8-25d876d21775" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:17:25.259750 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:25.259714    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf" podUID="5ea4957c-61b3-4888-a197-db7ddefbd6ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:17:34.251910 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:34.251868    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk" podUID="55e57a98-c38f-4ea2-b8e8-25d876d21775" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:17:35.260516 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:35.260477    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf" podUID="5ea4957c-61b3-4888-a197-db7ddefbd6ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:17:44.251579 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:44.251542    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk" podUID="55e57a98-c38f-4ea2-b8e8-25d876d21775" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:17:45.259637 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:45.259598    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf" podUID="5ea4957c-61b3-4888-a197-db7ddefbd6ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:17:54.251912 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:54.251870    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk" podUID="55e57a98-c38f-4ea2-b8e8-25d876d21775" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:17:55.259954 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:17:55.259911    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf" podUID="5ea4957c-61b3-4888-a197-db7ddefbd6ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:18:04.251802 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:04.251761    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk" podUID="55e57a98-c38f-4ea2-b8e8-25d876d21775" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:18:05.259870 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:05.259832    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf" podUID="5ea4957c-61b3-4888-a197-db7ddefbd6ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:18:14.251767 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:14.251730    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk" podUID="55e57a98-c38f-4ea2-b8e8-25d876d21775" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:18:15.259759 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:15.259722    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf" podUID="5ea4957c-61b3-4888-a197-db7ddefbd6ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:18:18.044606 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:18.044564    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf" podUID="5ea4957c-61b3-4888-a197-db7ddefbd6ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:18:24.252301 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:24.252271    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk"
Sep 17 18:18:28.049657 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:28.049592    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf"
Sep 17 18:18:31.887853 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:31.887818    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf"]
Sep 17 18:18:31.888328 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:31.888119    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf" podUID="5ea4957c-61b3-4888-a197-db7ddefbd6ea" containerName="kserve-container" containerID="cri-o://e9a245cbaeb21aaf1f6979902e5ec01a4d601d79f2a1c351fcb621f61ca20571" gracePeriod=30
Sep 17 18:18:31.888328 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:31.888198    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf" podUID="5ea4957c-61b3-4888-a197-db7ddefbd6ea" containerName="kube-rbac-proxy" containerID="cri-o://11f402ccfd5b099ef00560806ee6d1ee5d5ab5c522c2e05eeed1e620f71ff4c8" gracePeriod=30
Sep 17 18:18:31.892553 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:31.892523    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk"]
Sep 17 18:18:31.892809 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:31.892764    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk" podUID="55e57a98-c38f-4ea2-b8e8-25d876d21775" containerName="kserve-container" containerID="cri-o://a44022d4dffdb44f17df05243d980bce89f9afb8c115be82d245a66c47a3ca2c" gracePeriod=30
Sep 17 18:18:31.892903 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:31.892866    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk" podUID="55e57a98-c38f-4ea2-b8e8-25d876d21775" containerName="kube-rbac-proxy" containerID="cri-o://24f7c37c945da1198a971e7f0f2d72a8fb0bfeca20b2994ef828f9bc744526eb" gracePeriod=30
Sep 17 18:18:32.594716 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:32.594683    2581 generic.go:356] "Generic (PLEG): container finished" podID="55e57a98-c38f-4ea2-b8e8-25d876d21775" containerID="24f7c37c945da1198a971e7f0f2d72a8fb0bfeca20b2994ef828f9bc744526eb" exitCode=2
Sep 17 18:18:32.594918 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:32.594763    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk" event={"ID":"55e57a98-c38f-4ea2-b8e8-25d876d21775","Type":"ContainerDied","Data":"24f7c37c945da1198a971e7f0f2d72a8fb0bfeca20b2994ef828f9bc744526eb"}
Sep 17 18:18:32.597074 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:32.597041    2581 generic.go:356] "Generic (PLEG): container finished" podID="5ea4957c-61b3-4888-a197-db7ddefbd6ea" containerID="11f402ccfd5b099ef00560806ee6d1ee5d5ab5c522c2e05eeed1e620f71ff4c8" exitCode=2
Sep 17 18:18:32.597280 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:32.597112    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf" event={"ID":"5ea4957c-61b3-4888-a197-db7ddefbd6ea","Type":"ContainerDied","Data":"11f402ccfd5b099ef00560806ee6d1ee5d5ab5c522c2e05eeed1e620f71ff4c8"}
Sep 17 18:18:34.247870 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:34.247815    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk" podUID="55e57a98-c38f-4ea2-b8e8-25d876d21775" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 17 18:18:34.252302 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:34.252267    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk" podUID="55e57a98-c38f-4ea2-b8e8-25d876d21775" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:18:35.254384 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:35.254332    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf" podUID="5ea4957c-61b3-4888-a197-db7ddefbd6ea" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 17 18:18:37.140956 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.140919    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv"]
Sep 17 18:18:37.141530 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.141506    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="kserve-container"
Sep 17 18:18:37.141610 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.141556    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="kserve-container"
Sep 17 18:18:37.141610 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.141573    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="storage-initializer"
Sep 17 18:18:37.141610 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.141582    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="storage-initializer"
Sep 17 18:18:37.141760 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.141613    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="kube-rbac-proxy"
Sep 17 18:18:37.141760 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.141621    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="kube-rbac-proxy"
Sep 17 18:18:37.141760 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.141743    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="kube-rbac-proxy"
Sep 17 18:18:37.141760 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.141755    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ea0501d-7e40-4ff8-946d-cf5bc861bf9c" containerName="kserve-container"
Sep 17 18:18:37.153859 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.153822    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv"
Sep 17 18:18:37.156362 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.156321    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 18:18:37.156576 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.156513    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 18:18:37.156932 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.156905    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv"]
Sep 17 18:18:37.158387 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.158361    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7877c757df-czzhv\" (UID: \"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv"
Sep 17 18:18:37.158523 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.158451    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-proxy-tls\") pod \"isvc-canary-promote-predictor-7877c757df-czzhv\" (UID: \"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv"
Sep 17 18:18:37.158523 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.158502    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-cabundle-cert\") pod \"isvc-canary-promote-predictor-7877c757df-czzhv\" (UID: \"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv"
Sep 17 18:18:37.158698 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.158589    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7877c757df-czzhv\" (UID: \"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv"
Sep 17 18:18:37.158764 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.158691    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l7fq\" (UniqueName: \"kubernetes.io/projected/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-kube-api-access-5l7fq\") pod \"isvc-canary-promote-predictor-7877c757df-czzhv\" (UID: \"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv"
Sep 17 18:18:37.260001 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.259964    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-proxy-tls\") pod \"isvc-canary-promote-predictor-7877c757df-czzhv\" (UID: \"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv"
Sep 17 18:18:37.260208 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.260015    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-cabundle-cert\") pod \"isvc-canary-promote-predictor-7877c757df-czzhv\" (UID: \"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv"
Sep 17 18:18:37.260208 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.260066    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7877c757df-czzhv\" (UID: \"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv"
Sep 17 18:18:37.260208 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.260098    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l7fq\" (UniqueName: \"kubernetes.io/projected/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-kube-api-access-5l7fq\") pod \"isvc-canary-promote-predictor-7877c757df-czzhv\" (UID: \"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv"
Sep 17 18:18:37.260208 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:18:37.260126    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 17 18:18:37.260208 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.260175    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7877c757df-czzhv\" (UID: \"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv"
Sep 17 18:18:37.260502 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:18:37.260225    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-proxy-tls podName:5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3 nodeName:}" failed. No retries permitted until 2026-09-17 18:18:37.760201105 +0000 UTC m=+1299.860664918 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-proxy-tls") pod "isvc-canary-promote-predictor-7877c757df-czzhv" (UID: "5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 17 18:18:37.260572 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.260508    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7877c757df-czzhv\" (UID: \"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv"
Sep 17 18:18:37.260830 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.260796    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-cabundle-cert\") pod \"isvc-canary-promote-predictor-7877c757df-czzhv\" (UID: \"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv"
Sep 17 18:18:37.260931 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.260872    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7877c757df-czzhv\" (UID: \"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv"
Sep 17 18:18:37.268657 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.268623    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l7fq\" (UniqueName: \"kubernetes.io/projected/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-kube-api-access-5l7fq\") pod \"isvc-canary-promote-predictor-7877c757df-czzhv\" (UID: \"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv"
Sep 17 18:18:37.338946 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.338918    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk"
Sep 17 18:18:37.360871 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.360837    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55e57a98-c38f-4ea2-b8e8-25d876d21775-kserve-provision-location\") pod \"55e57a98-c38f-4ea2-b8e8-25d876d21775\" (UID: \"55e57a98-c38f-4ea2-b8e8-25d876d21775\") "
Sep 17 18:18:37.361082 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.360951    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55e57a98-c38f-4ea2-b8e8-25d876d21775-kube-rbac-proxy-sar-config\") pod \"55e57a98-c38f-4ea2-b8e8-25d876d21775\" (UID: \"55e57a98-c38f-4ea2-b8e8-25d876d21775\") "
Sep 17 18:18:37.361082 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.360990    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55e57a98-c38f-4ea2-b8e8-25d876d21775-proxy-tls\") pod \"55e57a98-c38f-4ea2-b8e8-25d876d21775\" (UID: \"55e57a98-c38f-4ea2-b8e8-25d876d21775\") "
Sep 17 18:18:37.361082 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.361022    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t962c\" (UniqueName: \"kubernetes.io/projected/55e57a98-c38f-4ea2-b8e8-25d876d21775-kube-api-access-t962c\") pod \"55e57a98-c38f-4ea2-b8e8-25d876d21775\" (UID: \"55e57a98-c38f-4ea2-b8e8-25d876d21775\") "
Sep 17 18:18:37.361082 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.361078    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55e57a98-c38f-4ea2-b8e8-25d876d21775-cabundle-cert\") pod \"55e57a98-c38f-4ea2-b8e8-25d876d21775\" (UID: \"55e57a98-c38f-4ea2-b8e8-25d876d21775\") "
Sep 17 18:18:37.361941 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.361832    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e57a98-c38f-4ea2-b8e8-25d876d21775-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "55e57a98-c38f-4ea2-b8e8-25d876d21775" (UID: "55e57a98-c38f-4ea2-b8e8-25d876d21775"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:37.362073 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.362018    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55e57a98-c38f-4ea2-b8e8-25d876d21775-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "55e57a98-c38f-4ea2-b8e8-25d876d21775" (UID: "55e57a98-c38f-4ea2-b8e8-25d876d21775"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:18:37.362233 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.362187    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55e57a98-c38f-4ea2-b8e8-25d876d21775-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "55e57a98-c38f-4ea2-b8e8-25d876d21775" (UID: "55e57a98-c38f-4ea2-b8e8-25d876d21775"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:18:37.364325 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.364296    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55e57a98-c38f-4ea2-b8e8-25d876d21775-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "55e57a98-c38f-4ea2-b8e8-25d876d21775" (UID: "55e57a98-c38f-4ea2-b8e8-25d876d21775"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:18:37.364410 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.364357    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e57a98-c38f-4ea2-b8e8-25d876d21775-kube-api-access-t962c" (OuterVolumeSpecName: "kube-api-access-t962c") pod "55e57a98-c38f-4ea2-b8e8-25d876d21775" (UID: "55e57a98-c38f-4ea2-b8e8-25d876d21775"). InnerVolumeSpecName "kube-api-access-t962c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:18:37.451924 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.451895    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf"
Sep 17 18:18:37.461960 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.461933    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ea4957c-61b3-4888-a197-db7ddefbd6ea-proxy-tls\") pod \"5ea4957c-61b3-4888-a197-db7ddefbd6ea\" (UID: \"5ea4957c-61b3-4888-a197-db7ddefbd6ea\") "
Sep 17 18:18:37.462140 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.462019    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7kdq\" (UniqueName: \"kubernetes.io/projected/5ea4957c-61b3-4888-a197-db7ddefbd6ea-kube-api-access-s7kdq\") pod \"5ea4957c-61b3-4888-a197-db7ddefbd6ea\" (UID: \"5ea4957c-61b3-4888-a197-db7ddefbd6ea\") "
Sep 17 18:18:37.462140 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.462059    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ea4957c-61b3-4888-a197-db7ddefbd6ea-kserve-provision-location\") pod \"5ea4957c-61b3-4888-a197-db7ddefbd6ea\" (UID: \"5ea4957c-61b3-4888-a197-db7ddefbd6ea\") "
Sep 17 18:18:37.462140 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.462090    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ea4957c-61b3-4888-a197-db7ddefbd6ea-cabundle-cert\") pod \"5ea4957c-61b3-4888-a197-db7ddefbd6ea\" (UID: \"5ea4957c-61b3-4888-a197-db7ddefbd6ea\") "
Sep 17 18:18:37.462140 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.462131    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ea4957c-61b3-4888-a197-db7ddefbd6ea-kube-rbac-proxy-sar-config\") pod \"5ea4957c-61b3-4888-a197-db7ddefbd6ea\" (UID: \"5ea4957c-61b3-4888-a197-db7ddefbd6ea\") "
Sep 17 18:18:37.462441 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.462423    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55e57a98-c38f-4ea2-b8e8-25d876d21775-cabundle-cert\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:18:37.462518 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.462431    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ea4957c-61b3-4888-a197-db7ddefbd6ea-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5ea4957c-61b3-4888-a197-db7ddefbd6ea" (UID: "5ea4957c-61b3-4888-a197-db7ddefbd6ea"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:37.462563 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.462449    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55e57a98-c38f-4ea2-b8e8-25d876d21775-kserve-provision-location\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:18:37.462563 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.462537    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea4957c-61b3-4888-a197-db7ddefbd6ea-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5ea4957c-61b3-4888-a197-db7ddefbd6ea" (UID: "5ea4957c-61b3-4888-a197-db7ddefbd6ea"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:18:37.462646 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.462559    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55e57a98-c38f-4ea2-b8e8-25d876d21775-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:18:37.462646 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.462579    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55e57a98-c38f-4ea2-b8e8-25d876d21775-proxy-tls\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:18:37.462646 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.462595    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t962c\" (UniqueName: \"kubernetes.io/projected/55e57a98-c38f-4ea2-b8e8-25d876d21775-kube-api-access-t962c\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:18:37.462646 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.462597    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea4957c-61b3-4888-a197-db7ddefbd6ea-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5ea4957c-61b3-4888-a197-db7ddefbd6ea" (UID: "5ea4957c-61b3-4888-a197-db7ddefbd6ea"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:18:37.464306 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.464282    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea4957c-61b3-4888-a197-db7ddefbd6ea-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5ea4957c-61b3-4888-a197-db7ddefbd6ea" (UID: "5ea4957c-61b3-4888-a197-db7ddefbd6ea"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:18:37.464578 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.464558    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea4957c-61b3-4888-a197-db7ddefbd6ea-kube-api-access-s7kdq" (OuterVolumeSpecName: "kube-api-access-s7kdq") pod "5ea4957c-61b3-4888-a197-db7ddefbd6ea" (UID: "5ea4957c-61b3-4888-a197-db7ddefbd6ea"). InnerVolumeSpecName "kube-api-access-s7kdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:18:37.563572 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.563539    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ea4957c-61b3-4888-a197-db7ddefbd6ea-kserve-provision-location\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:18:37.563572 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.563569    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ea4957c-61b3-4888-a197-db7ddefbd6ea-cabundle-cert\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:18:37.563572 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.563578    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ea4957c-61b3-4888-a197-db7ddefbd6ea-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:18:37.563824 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.563587    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ea4957c-61b3-4888-a197-db7ddefbd6ea-proxy-tls\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:18:37.563824 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.563597    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7kdq\" (UniqueName: \"kubernetes.io/projected/5ea4957c-61b3-4888-a197-db7ddefbd6ea-kube-api-access-s7kdq\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:18:37.618954 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.618917    2581 generic.go:356] "Generic (PLEG): container finished" podID="5ea4957c-61b3-4888-a197-db7ddefbd6ea" containerID="e9a245cbaeb21aaf1f6979902e5ec01a4d601d79f2a1c351fcb621f61ca20571" exitCode=0
Sep 17 18:18:37.619133 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.618992    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf"
Sep 17 18:18:37.619133 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.619007    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf" event={"ID":"5ea4957c-61b3-4888-a197-db7ddefbd6ea","Type":"ContainerDied","Data":"e9a245cbaeb21aaf1f6979902e5ec01a4d601d79f2a1c351fcb621f61ca20571"}
Sep 17 18:18:37.619133 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.619073    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf" event={"ID":"5ea4957c-61b3-4888-a197-db7ddefbd6ea","Type":"ContainerDied","Data":"d0c50f5a7e34a4a0c46e3fa63314114d737ad57d3b92298eb3e7708ccce91969"}
Sep 17 18:18:37.619133 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.619105    2581 scope.go:117] "RemoveContainer" containerID="11f402ccfd5b099ef00560806ee6d1ee5d5ab5c522c2e05eeed1e620f71ff4c8"
Sep 17 18:18:37.621136 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.621091    2581 generic.go:356] "Generic (PLEG): container finished" podID="55e57a98-c38f-4ea2-b8e8-25d876d21775" containerID="a44022d4dffdb44f17df05243d980bce89f9afb8c115be82d245a66c47a3ca2c" exitCode=0
Sep 17 18:18:37.621278 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.621175    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk"
Sep 17 18:18:37.621278 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.621188    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk" event={"ID":"55e57a98-c38f-4ea2-b8e8-25d876d21775","Type":"ContainerDied","Data":"a44022d4dffdb44f17df05243d980bce89f9afb8c115be82d245a66c47a3ca2c"}
Sep 17 18:18:37.621278 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.621238    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk" event={"ID":"55e57a98-c38f-4ea2-b8e8-25d876d21775","Type":"ContainerDied","Data":"43b1f9cd96eac30e4e713c9aee6cbd5a19147ffc3b41f974c436094b93c770cc"}
Sep 17 18:18:37.628101 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.628075    2581 scope.go:117] "RemoveContainer" containerID="e9a245cbaeb21aaf1f6979902e5ec01a4d601d79f2a1c351fcb621f61ca20571"
Sep 17 18:18:37.636346 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.636326    2581 scope.go:117] "RemoveContainer" containerID="208784c947986ae567a826292a93d0b2ac8fc4b53f49a44037f7b19971f92c66"
Sep 17 18:18:37.641708 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.641683    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf"]
Sep 17 18:18:37.644982 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.644962    2581 scope.go:117] "RemoveContainer" containerID="11f402ccfd5b099ef00560806ee6d1ee5d5ab5c522c2e05eeed1e620f71ff4c8"
Sep 17 18:18:37.645466 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:18:37.645429    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11f402ccfd5b099ef00560806ee6d1ee5d5ab5c522c2e05eeed1e620f71ff4c8\": container with ID starting with 11f402ccfd5b099ef00560806ee6d1ee5d5ab5c522c2e05eeed1e620f71ff4c8 not found: ID does not exist" containerID="11f402ccfd5b099ef00560806ee6d1ee5d5ab5c522c2e05eeed1e620f71ff4c8"
Sep 17 18:18:37.645576 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.645467    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11f402ccfd5b099ef00560806ee6d1ee5d5ab5c522c2e05eeed1e620f71ff4c8"} err="failed to get container status \"11f402ccfd5b099ef00560806ee6d1ee5d5ab5c522c2e05eeed1e620f71ff4c8\": rpc error: code = NotFound desc = could not find container \"11f402ccfd5b099ef00560806ee6d1ee5d5ab5c522c2e05eeed1e620f71ff4c8\": container with ID starting with 11f402ccfd5b099ef00560806ee6d1ee5d5ab5c522c2e05eeed1e620f71ff4c8 not found: ID does not exist"
Sep 17 18:18:37.645576 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.645500    2581 scope.go:117] "RemoveContainer" containerID="e9a245cbaeb21aaf1f6979902e5ec01a4d601d79f2a1c351fcb621f61ca20571"
Sep 17 18:18:37.646199 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:18:37.646176    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9a245cbaeb21aaf1f6979902e5ec01a4d601d79f2a1c351fcb621f61ca20571\": container with ID starting with e9a245cbaeb21aaf1f6979902e5ec01a4d601d79f2a1c351fcb621f61ca20571 not found: ID does not exist" containerID="e9a245cbaeb21aaf1f6979902e5ec01a4d601d79f2a1c351fcb621f61ca20571"
Sep 17 18:18:37.646299 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.646205    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a245cbaeb21aaf1f6979902e5ec01a4d601d79f2a1c351fcb621f61ca20571"} err="failed to get container status \"e9a245cbaeb21aaf1f6979902e5ec01a4d601d79f2a1c351fcb621f61ca20571\": rpc error: code = NotFound desc = could not find container \"e9a245cbaeb21aaf1f6979902e5ec01a4d601d79f2a1c351fcb621f61ca20571\": container with ID starting with e9a245cbaeb21aaf1f6979902e5ec01a4d601d79f2a1c351fcb621f61ca20571 not found: ID does not exist"
Sep 17 18:18:37.646299 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.646222    2581 scope.go:117] "RemoveContainer" containerID="208784c947986ae567a826292a93d0b2ac8fc4b53f49a44037f7b19971f92c66"
Sep 17 18:18:37.646525 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:18:37.646505    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"208784c947986ae567a826292a93d0b2ac8fc4b53f49a44037f7b19971f92c66\": container with ID starting with 208784c947986ae567a826292a93d0b2ac8fc4b53f49a44037f7b19971f92c66 not found: ID does not exist" containerID="208784c947986ae567a826292a93d0b2ac8fc4b53f49a44037f7b19971f92c66"
Sep 17 18:18:37.646568 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.646531    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"208784c947986ae567a826292a93d0b2ac8fc4b53f49a44037f7b19971f92c66"} err="failed to get container status \"208784c947986ae567a826292a93d0b2ac8fc4b53f49a44037f7b19971f92c66\": rpc error: code = NotFound desc = could not find container \"208784c947986ae567a826292a93d0b2ac8fc4b53f49a44037f7b19971f92c66\": container with ID starting with 208784c947986ae567a826292a93d0b2ac8fc4b53f49a44037f7b19971f92c66 not found: ID does not exist"
Sep 17 18:18:37.646568 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.646548    2581 scope.go:117] "RemoveContainer" containerID="24f7c37c945da1198a971e7f0f2d72a8fb0bfeca20b2994ef828f9bc744526eb"
Sep 17 18:18:37.647840 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.647817    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5bb585469b-kqcgf"]
Sep 17 18:18:37.654917 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.654892    2581 scope.go:117] "RemoveContainer" containerID="a44022d4dffdb44f17df05243d980bce89f9afb8c115be82d245a66c47a3ca2c"
Sep 17 18:18:37.657183 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.657136    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk"]
Sep 17 18:18:37.660730 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.660705    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-kjzjk"]
Sep 17 18:18:37.663700 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.663679    2581 scope.go:117] "RemoveContainer" containerID="0f298e368cc8c3ac27625c38092dbb28223bb45fc5dc52c9691a807f524b6028"
Sep 17 18:18:37.671759 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.671741    2581 scope.go:117] "RemoveContainer" containerID="24f7c37c945da1198a971e7f0f2d72a8fb0bfeca20b2994ef828f9bc744526eb"
Sep 17 18:18:37.672058 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:18:37.672038    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24f7c37c945da1198a971e7f0f2d72a8fb0bfeca20b2994ef828f9bc744526eb\": container with ID starting with 24f7c37c945da1198a971e7f0f2d72a8fb0bfeca20b2994ef828f9bc744526eb not found: ID does not exist" containerID="24f7c37c945da1198a971e7f0f2d72a8fb0bfeca20b2994ef828f9bc744526eb"
Sep 17 18:18:37.672135 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.672069    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24f7c37c945da1198a971e7f0f2d72a8fb0bfeca20b2994ef828f9bc744526eb"} err="failed to get container status \"24f7c37c945da1198a971e7f0f2d72a8fb0bfeca20b2994ef828f9bc744526eb\": rpc error: code = NotFound desc = could not find container \"24f7c37c945da1198a971e7f0f2d72a8fb0bfeca20b2994ef828f9bc744526eb\": container with ID starting with 24f7c37c945da1198a971e7f0f2d72a8fb0bfeca20b2994ef828f9bc744526eb not found: ID does not exist"
Sep 17 18:18:37.672135 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.672096    2581 scope.go:117] "RemoveContainer" containerID="a44022d4dffdb44f17df05243d980bce89f9afb8c115be82d245a66c47a3ca2c"
Sep 17 18:18:37.672402 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:18:37.672386    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a44022d4dffdb44f17df05243d980bce89f9afb8c115be82d245a66c47a3ca2c\": container with ID starting with a44022d4dffdb44f17df05243d980bce89f9afb8c115be82d245a66c47a3ca2c not found: ID does not exist" containerID="a44022d4dffdb44f17df05243d980bce89f9afb8c115be82d245a66c47a3ca2c"
Sep 17 18:18:37.672442 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.672410    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a44022d4dffdb44f17df05243d980bce89f9afb8c115be82d245a66c47a3ca2c"} err="failed to get container status \"a44022d4dffdb44f17df05243d980bce89f9afb8c115be82d245a66c47a3ca2c\": rpc error: code = NotFound desc = could not find container \"a44022d4dffdb44f17df05243d980bce89f9afb8c115be82d245a66c47a3ca2c\": container with ID starting with a44022d4dffdb44f17df05243d980bce89f9afb8c115be82d245a66c47a3ca2c not found: ID does not exist"
Sep 17 18:18:37.672442 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.672430    2581 scope.go:117] "RemoveContainer" containerID="0f298e368cc8c3ac27625c38092dbb28223bb45fc5dc52c9691a807f524b6028"
Sep 17 18:18:37.672704 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:18:37.672687    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f298e368cc8c3ac27625c38092dbb28223bb45fc5dc52c9691a807f524b6028\": container with ID starting with 0f298e368cc8c3ac27625c38092dbb28223bb45fc5dc52c9691a807f524b6028 not found: ID does not exist" containerID="0f298e368cc8c3ac27625c38092dbb28223bb45fc5dc52c9691a807f524b6028"
Sep 17 18:18:37.672744 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.672710    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f298e368cc8c3ac27625c38092dbb28223bb45fc5dc52c9691a807f524b6028"} err="failed to get container status \"0f298e368cc8c3ac27625c38092dbb28223bb45fc5dc52c9691a807f524b6028\": rpc error: code = NotFound desc = could not find container \"0f298e368cc8c3ac27625c38092dbb28223bb45fc5dc52c9691a807f524b6028\": container with ID starting with 0f298e368cc8c3ac27625c38092dbb28223bb45fc5dc52c9691a807f524b6028 not found: ID does not exist"
Sep 17 18:18:37.764977 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.764854    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-proxy-tls\") pod \"isvc-canary-promote-predictor-7877c757df-czzhv\" (UID: \"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv"
Sep 17 18:18:37.767531 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.767511    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-proxy-tls\") pod \"isvc-canary-promote-predictor-7877c757df-czzhv\" (UID: \"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv"
Sep 17 18:18:37.804500 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:37.804462    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv"
Sep 17 18:18:38.049510 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:38.049421    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55e57a98-c38f-4ea2-b8e8-25d876d21775" path="/var/lib/kubelet/pods/55e57a98-c38f-4ea2-b8e8-25d876d21775/volumes"
Sep 17 18:18:38.050016 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:38.049995    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ea4957c-61b3-4888-a197-db7ddefbd6ea" path="/var/lib/kubelet/pods/5ea4957c-61b3-4888-a197-db7ddefbd6ea/volumes"
Sep 17 18:18:38.079974 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:38.079939    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv"]
Sep 17 18:18:38.083423 ip-10-0-134-209 kubenswrapper[2581]: W0917 18:18:38.083394    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e6a2de7_9e3c_4387_9ec4_127ed99cb4d3.slice/crio-af49c10b38b7e9bc9c93e961d9c3071e0be87d052674c5c78322af39ced11076 WatchSource:0}: Error finding container af49c10b38b7e9bc9c93e961d9c3071e0be87d052674c5c78322af39ced11076: Status 404 returned error can't find the container with id af49c10b38b7e9bc9c93e961d9c3071e0be87d052674c5c78322af39ced11076
Sep 17 18:18:38.629383 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:38.629341    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv" event={"ID":"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3","Type":"ContainerStarted","Data":"27b3bf7829e3811929c460a96de8be003e5e8607c932af2a52c3342fa7a29386"}
Sep 17 18:18:38.629872 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:38.629402    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv" event={"ID":"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3","Type":"ContainerStarted","Data":"af49c10b38b7e9bc9c93e961d9c3071e0be87d052674c5c78322af39ced11076"}
Sep 17 18:18:42.648851 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:42.648812    2581 generic.go:356] "Generic (PLEG): container finished" podID="5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3" containerID="27b3bf7829e3811929c460a96de8be003e5e8607c932af2a52c3342fa7a29386" exitCode=0
Sep 17 18:18:42.649282 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:42.648887    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv" event={"ID":"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3","Type":"ContainerDied","Data":"27b3bf7829e3811929c460a96de8be003e5e8607c932af2a52c3342fa7a29386"}
Sep 17 18:18:43.656338 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:43.656296    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv" event={"ID":"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3","Type":"ContainerStarted","Data":"a414ed7906def10312ff164897348d3f7ed4cd511fd7d3e4e4c2f4ece2b69af0"}
Sep 17 18:18:43.656832 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:43.656371    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv" event={"ID":"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3","Type":"ContainerStarted","Data":"3c209bda8f3063376425e2e8cdb5f3cf50c06ee67369b0fc66675c85ccec5455"}
Sep 17 18:18:43.656832 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:43.656590    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv"
Sep 17 18:18:43.675331 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:43.675271    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv" podStartSLOduration=2.83351743 podStartE2EDuration="6.675251287s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.841733857s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:18:43.673601262 +0000 UTC m=+1305.774065082" watchObservedRunningTime="2026-09-17 18:18:43.675251287 +0000 UTC m=+1305.775715106"
Sep 17 18:18:44.661132 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:44.661092    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv"
Sep 17 18:18:44.662519 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:44.662483    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv" podUID="5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 18:18:45.665560 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:45.665517    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv" podUID="5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 18:18:50.670274 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:50.670243    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv"
Sep 17 18:18:50.670962 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:18:50.670932    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv" podUID="5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 18:19:00.671266 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:19:00.671227    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv" podUID="5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 18:19:10.671382 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:19:10.671276    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv" podUID="5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 18:19:20.671629 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:19:20.671588    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv" podUID="5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 18:19:30.671792 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:19:30.671755    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv" podUID="5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 18:19:40.671107 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:19:40.671066    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv" podUID="5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 18:19:50.671415 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:19:50.671373    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv" podUID="5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 18:20:00.671564 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:00.671535    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv"
Sep 17 18:20:07.179320 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.179290    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv"]
Sep 17 18:20:07.179786 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.179529    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv" podUID="5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3" containerName="kserve-container" containerID="cri-o://3c209bda8f3063376425e2e8cdb5f3cf50c06ee67369b0fc66675c85ccec5455" gracePeriod=30
Sep 17 18:20:07.179786 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.179603    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv" podUID="5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3" containerName="kube-rbac-proxy" containerID="cri-o://a414ed7906def10312ff164897348d3f7ed4cd511fd7d3e4e4c2f4ece2b69af0" gracePeriod=30
Sep 17 18:20:07.251710 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.251676    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg"]
Sep 17 18:20:07.252007 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.251988    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55e57a98-c38f-4ea2-b8e8-25d876d21775" containerName="kube-rbac-proxy"
Sep 17 18:20:07.252007 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.252004    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="55e57a98-c38f-4ea2-b8e8-25d876d21775" containerName="kube-rbac-proxy"
Sep 17 18:20:07.252099 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.252022    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55e57a98-c38f-4ea2-b8e8-25d876d21775" containerName="kserve-container"
Sep 17 18:20:07.252099 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.252027    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="55e57a98-c38f-4ea2-b8e8-25d876d21775" containerName="kserve-container"
Sep 17 18:20:07.252099 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.252037    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55e57a98-c38f-4ea2-b8e8-25d876d21775" containerName="storage-initializer"
Sep 17 18:20:07.252099 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.252044    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="55e57a98-c38f-4ea2-b8e8-25d876d21775" containerName="storage-initializer"
Sep 17 18:20:07.252099 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.252050    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ea4957c-61b3-4888-a197-db7ddefbd6ea" containerName="storage-initializer"
Sep 17 18:20:07.252099 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.252055    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ea4957c-61b3-4888-a197-db7ddefbd6ea" containerName="storage-initializer"
Sep 17 18:20:07.252099 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.252062    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ea4957c-61b3-4888-a197-db7ddefbd6ea" containerName="kserve-container"
Sep 17 18:20:07.252099 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.252066    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ea4957c-61b3-4888-a197-db7ddefbd6ea" containerName="kserve-container"
Sep 17 18:20:07.252099 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.252075    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ea4957c-61b3-4888-a197-db7ddefbd6ea" containerName="kube-rbac-proxy"
Sep 17 18:20:07.252099 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.252079    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ea4957c-61b3-4888-a197-db7ddefbd6ea" containerName="kube-rbac-proxy"
Sep 17 18:20:07.252419 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.252141    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="55e57a98-c38f-4ea2-b8e8-25d876d21775" containerName="kube-rbac-proxy"
Sep 17 18:20:07.252419 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.252167    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="55e57a98-c38f-4ea2-b8e8-25d876d21775" containerName="kserve-container"
Sep 17 18:20:07.252419 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.252179    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ea4957c-61b3-4888-a197-db7ddefbd6ea" containerName="kube-rbac-proxy"
Sep 17 18:20:07.252419 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.252186    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ea4957c-61b3-4888-a197-db7ddefbd6ea" containerName="kserve-container"
Sep 17 18:20:07.255363 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.255347    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg"
Sep 17 18:20:07.257496 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.257477    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 18:20:07.264634 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.264610    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg"]
Sep 17 18:20:07.402349 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.402321    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5191a2c0-f6c5-4f46-9264-59f528fc57e6-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg\" (UID: \"5191a2c0-f6c5-4f46-9264-59f528fc57e6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg"
Sep 17 18:20:07.402457 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.402356    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5191a2c0-f6c5-4f46-9264-59f528fc57e6-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg\" (UID: \"5191a2c0-f6c5-4f46-9264-59f528fc57e6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg"
Sep 17 18:20:07.402457 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.402375    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5191a2c0-f6c5-4f46-9264-59f528fc57e6-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg\" (UID: \"5191a2c0-f6c5-4f46-9264-59f528fc57e6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg"
Sep 17 18:20:07.402457 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.402403    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjtgt\" (UniqueName: \"kubernetes.io/projected/5191a2c0-f6c5-4f46-9264-59f528fc57e6-kube-api-access-wjtgt\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg\" (UID: \"5191a2c0-f6c5-4f46-9264-59f528fc57e6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg"
Sep 17 18:20:07.402573 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.402478    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5191a2c0-f6c5-4f46-9264-59f528fc57e6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg\" (UID: \"5191a2c0-f6c5-4f46-9264-59f528fc57e6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg"
Sep 17 18:20:07.503036 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.502971    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5191a2c0-f6c5-4f46-9264-59f528fc57e6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg\" (UID: \"5191a2c0-f6c5-4f46-9264-59f528fc57e6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg"
Sep 17 18:20:07.503036 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.503018    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5191a2c0-f6c5-4f46-9264-59f528fc57e6-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg\" (UID: \"5191a2c0-f6c5-4f46-9264-59f528fc57e6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg"
Sep 17 18:20:07.503200 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.503046    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5191a2c0-f6c5-4f46-9264-59f528fc57e6-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg\" (UID: \"5191a2c0-f6c5-4f46-9264-59f528fc57e6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg"
Sep 17 18:20:07.503200 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.503070    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5191a2c0-f6c5-4f46-9264-59f528fc57e6-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg\" (UID: \"5191a2c0-f6c5-4f46-9264-59f528fc57e6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg"
Sep 17 18:20:07.503200 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.503096    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjtgt\" (UniqueName: \"kubernetes.io/projected/5191a2c0-f6c5-4f46-9264-59f528fc57e6-kube-api-access-wjtgt\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg\" (UID: \"5191a2c0-f6c5-4f46-9264-59f528fc57e6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg"
Sep 17 18:20:07.503379 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.503357    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5191a2c0-f6c5-4f46-9264-59f528fc57e6-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg\" (UID: \"5191a2c0-f6c5-4f46-9264-59f528fc57e6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg"
Sep 17 18:20:07.503717 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.503695    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5191a2c0-f6c5-4f46-9264-59f528fc57e6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg\" (UID: \"5191a2c0-f6c5-4f46-9264-59f528fc57e6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg"
Sep 17 18:20:07.503797 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.503750    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5191a2c0-f6c5-4f46-9264-59f528fc57e6-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg\" (UID: \"5191a2c0-f6c5-4f46-9264-59f528fc57e6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg"
Sep 17 18:20:07.505519 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.505501    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5191a2c0-f6c5-4f46-9264-59f528fc57e6-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg\" (UID: \"5191a2c0-f6c5-4f46-9264-59f528fc57e6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg"
Sep 17 18:20:07.511480 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.511460    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjtgt\" (UniqueName: \"kubernetes.io/projected/5191a2c0-f6c5-4f46-9264-59f528fc57e6-kube-api-access-wjtgt\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg\" (UID: \"5191a2c0-f6c5-4f46-9264-59f528fc57e6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg"
Sep 17 18:20:07.566064 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.566040    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg"
Sep 17 18:20:07.841433 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.841409    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg"]
Sep 17 18:20:07.843784 ip-10-0-134-209 kubenswrapper[2581]: W0917 18:20:07.843758    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5191a2c0_f6c5_4f46_9264_59f528fc57e6.slice/crio-c0d457779a9e10ef9e769c8429cf0dd5bf56089846afc49e7007ce3a47a873d0 WatchSource:0}: Error finding container c0d457779a9e10ef9e769c8429cf0dd5bf56089846afc49e7007ce3a47a873d0: Status 404 returned error can't find the container with id c0d457779a9e10ef9e769c8429cf0dd5bf56089846afc49e7007ce3a47a873d0
Sep 17 18:20:07.991462 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.991428    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg" event={"ID":"5191a2c0-f6c5-4f46-9264-59f528fc57e6","Type":"ContainerStarted","Data":"53447951aa544f36c94db313b1063536b42363078beb673530d23a2fc60d0dc2"}
Sep 17 18:20:07.991636 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.991495    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg" event={"ID":"5191a2c0-f6c5-4f46-9264-59f528fc57e6","Type":"ContainerStarted","Data":"c0d457779a9e10ef9e769c8429cf0dd5bf56089846afc49e7007ce3a47a873d0"}
Sep 17 18:20:07.993710 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.993682    2581 generic.go:356] "Generic (PLEG): container finished" podID="5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3" containerID="a414ed7906def10312ff164897348d3f7ed4cd511fd7d3e4e4c2f4ece2b69af0" exitCode=2
Sep 17 18:20:07.993813 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:07.993739    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv" event={"ID":"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3","Type":"ContainerDied","Data":"a414ed7906def10312ff164897348d3f7ed4cd511fd7d3e4e4c2f4ece2b69af0"}
Sep 17 18:20:10.666046 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:10.666006    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv" podUID="5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 17 18:20:10.671715 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:10.671689    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv" podUID="5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 18:20:11.624379 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:11.624356    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv"
Sep 17 18:20:11.736658 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:11.736632    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5l7fq\" (UniqueName: \"kubernetes.io/projected/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-kube-api-access-5l7fq\") pod \"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3\" (UID: \"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3\") "
Sep 17 18:20:11.737041 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:11.736670    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-kserve-provision-location\") pod \"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3\" (UID: \"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3\") "
Sep 17 18:20:11.737041 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:11.736707    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-cabundle-cert\") pod \"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3\" (UID: \"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3\") "
Sep 17 18:20:11.737041 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:11.736737    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-kube-rbac-proxy-sar-config\") pod \"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3\" (UID: \"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3\") "
Sep 17 18:20:11.737041 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:11.736773    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-proxy-tls\") pod \"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3\" (UID: \"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3\") "
Sep 17 18:20:11.737041 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:11.736985    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3" (UID: "5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:11.737335 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:11.737093    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3" (UID: "5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:20:11.737335 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:11.737118    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3" (UID: "5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:20:11.738727 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:11.738704    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-kube-api-access-5l7fq" (OuterVolumeSpecName: "kube-api-access-5l7fq") pod "5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3" (UID: "5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3"). InnerVolumeSpecName "kube-api-access-5l7fq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:20:11.738901 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:11.738882    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3" (UID: "5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:20:11.837541 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:11.837509    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l7fq\" (UniqueName: \"kubernetes.io/projected/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-kube-api-access-5l7fq\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:20:11.837541 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:11.837537    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-kserve-provision-location\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:20:11.837705 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:11.837551    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-cabundle-cert\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:20:11.837705 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:11.837563    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:20:11.837705 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:11.837577    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3-proxy-tls\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:20:12.011285 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:12.011212    2581 generic.go:356] "Generic (PLEG): container finished" podID="5191a2c0-f6c5-4f46-9264-59f528fc57e6" containerID="53447951aa544f36c94db313b1063536b42363078beb673530d23a2fc60d0dc2" exitCode=0
Sep 17 18:20:12.011419 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:12.011289    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg" event={"ID":"5191a2c0-f6c5-4f46-9264-59f528fc57e6","Type":"ContainerDied","Data":"53447951aa544f36c94db313b1063536b42363078beb673530d23a2fc60d0dc2"}
Sep 17 18:20:12.013328 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:12.013306    2581 generic.go:356] "Generic (PLEG): container finished" podID="5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3" containerID="3c209bda8f3063376425e2e8cdb5f3cf50c06ee67369b0fc66675c85ccec5455" exitCode=0
Sep 17 18:20:12.013431 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:12.013358    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv"
Sep 17 18:20:12.013490 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:12.013362    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv" event={"ID":"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3","Type":"ContainerDied","Data":"3c209bda8f3063376425e2e8cdb5f3cf50c06ee67369b0fc66675c85ccec5455"}
Sep 17 18:20:12.013553 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:12.013488    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv" event={"ID":"5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3","Type":"ContainerDied","Data":"af49c10b38b7e9bc9c93e961d9c3071e0be87d052674c5c78322af39ced11076"}
Sep 17 18:20:12.013553 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:12.013520    2581 scope.go:117] "RemoveContainer" containerID="a414ed7906def10312ff164897348d3f7ed4cd511fd7d3e4e4c2f4ece2b69af0"
Sep 17 18:20:12.023801 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:12.023787    2581 scope.go:117] "RemoveContainer" containerID="3c209bda8f3063376425e2e8cdb5f3cf50c06ee67369b0fc66675c85ccec5455"
Sep 17 18:20:12.031573 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:12.031554    2581 scope.go:117] "RemoveContainer" containerID="27b3bf7829e3811929c460a96de8be003e5e8607c932af2a52c3342fa7a29386"
Sep 17 18:20:12.042241 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:12.042221    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv"]
Sep 17 18:20:12.049294 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:12.049274    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-czzhv"]
Sep 17 18:20:12.055258 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:12.055242    2581 scope.go:117] "RemoveContainer" containerID="a414ed7906def10312ff164897348d3f7ed4cd511fd7d3e4e4c2f4ece2b69af0"
Sep 17 18:20:12.055656 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:20:12.055634    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a414ed7906def10312ff164897348d3f7ed4cd511fd7d3e4e4c2f4ece2b69af0\": container with ID starting with a414ed7906def10312ff164897348d3f7ed4cd511fd7d3e4e4c2f4ece2b69af0 not found: ID does not exist" containerID="a414ed7906def10312ff164897348d3f7ed4cd511fd7d3e4e4c2f4ece2b69af0"
Sep 17 18:20:12.055736 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:12.055663    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a414ed7906def10312ff164897348d3f7ed4cd511fd7d3e4e4c2f4ece2b69af0"} err="failed to get container status \"a414ed7906def10312ff164897348d3f7ed4cd511fd7d3e4e4c2f4ece2b69af0\": rpc error: code = NotFound desc = could not find container \"a414ed7906def10312ff164897348d3f7ed4cd511fd7d3e4e4c2f4ece2b69af0\": container with ID starting with a414ed7906def10312ff164897348d3f7ed4cd511fd7d3e4e4c2f4ece2b69af0 not found: ID does not exist"
Sep 17 18:20:12.055736 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:12.055679    2581 scope.go:117] "RemoveContainer" containerID="3c209bda8f3063376425e2e8cdb5f3cf50c06ee67369b0fc66675c85ccec5455"
Sep 17 18:20:12.055985 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:20:12.055970    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c209bda8f3063376425e2e8cdb5f3cf50c06ee67369b0fc66675c85ccec5455\": container with ID starting with 3c209bda8f3063376425e2e8cdb5f3cf50c06ee67369b0fc66675c85ccec5455 not found: ID does not exist" containerID="3c209bda8f3063376425e2e8cdb5f3cf50c06ee67369b0fc66675c85ccec5455"
Sep 17 18:20:12.056032 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:12.055989    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c209bda8f3063376425e2e8cdb5f3cf50c06ee67369b0fc66675c85ccec5455"} err="failed to get container status \"3c209bda8f3063376425e2e8cdb5f3cf50c06ee67369b0fc66675c85ccec5455\": rpc error: code = NotFound desc = could not find container \"3c209bda8f3063376425e2e8cdb5f3cf50c06ee67369b0fc66675c85ccec5455\": container with ID starting with 3c209bda8f3063376425e2e8cdb5f3cf50c06ee67369b0fc66675c85ccec5455 not found: ID does not exist"
Sep 17 18:20:12.056032 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:12.056004    2581 scope.go:117] "RemoveContainer" containerID="27b3bf7829e3811929c460a96de8be003e5e8607c932af2a52c3342fa7a29386"
Sep 17 18:20:12.056327 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:20:12.056308    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27b3bf7829e3811929c460a96de8be003e5e8607c932af2a52c3342fa7a29386\": container with ID starting with 27b3bf7829e3811929c460a96de8be003e5e8607c932af2a52c3342fa7a29386 not found: ID does not exist" containerID="27b3bf7829e3811929c460a96de8be003e5e8607c932af2a52c3342fa7a29386"
Sep 17 18:20:12.056383 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:12.056330    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27b3bf7829e3811929c460a96de8be003e5e8607c932af2a52c3342fa7a29386"} err="failed to get container status \"27b3bf7829e3811929c460a96de8be003e5e8607c932af2a52c3342fa7a29386\": rpc error: code = NotFound desc = could not find container \"27b3bf7829e3811929c460a96de8be003e5e8607c932af2a52c3342fa7a29386\": container with ID starting with 27b3bf7829e3811929c460a96de8be003e5e8607c932af2a52c3342fa7a29386 not found: ID does not exist"
Sep 17 18:20:13.019634 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:13.019601    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg" event={"ID":"5191a2c0-f6c5-4f46-9264-59f528fc57e6","Type":"ContainerStarted","Data":"5391adf66bf92f6be5669b582142983f78c4289d4f3b151727178b33ce133dc3"}
Sep 17 18:20:13.020041 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:13.019662    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg" event={"ID":"5191a2c0-f6c5-4f46-9264-59f528fc57e6","Type":"ContainerStarted","Data":"da0661a59c926c40751bd5e674763afeb85095fc98a3ffd8d1104731b9cceccc"}
Sep 17 18:20:13.020041 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:13.019980    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg"
Sep 17 18:20:13.020199 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:13.020130    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg"
Sep 17 18:20:13.021630 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:13.021589    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg" podUID="5191a2c0-f6c5-4f46-9264-59f528fc57e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:20:13.043169 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:13.040033    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg" podStartSLOduration=2.24837478 podStartE2EDuration="6.040015439s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.791640659s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:20:13.037395467 +0000 UTC m=+1395.137859286" watchObservedRunningTime="2026-09-17 18:20:13.040015439 +0000 UTC m=+1395.140479259"
Sep 17 18:20:14.024996 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:14.024960    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg" podUID="5191a2c0-f6c5-4f46-9264-59f528fc57e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:20:14.050943 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:14.050915    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3" path="/var/lib/kubelet/pods/5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3/volumes"
Sep 17 18:20:19.029813 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:19.029783    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg"
Sep 17 18:20:19.030276 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:19.030248    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg" podUID="5191a2c0-f6c5-4f46-9264-59f528fc57e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:20:29.030907 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:29.030869    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg" podUID="5191a2c0-f6c5-4f46-9264-59f528fc57e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:20:39.030893 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:39.030853    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg" podUID="5191a2c0-f6c5-4f46-9264-59f528fc57e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:20:49.030785 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:49.030706    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg" podUID="5191a2c0-f6c5-4f46-9264-59f528fc57e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:20:59.030488 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:20:59.030447    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg" podUID="5191a2c0-f6c5-4f46-9264-59f528fc57e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:21:09.031255 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:09.031213    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg" podUID="5191a2c0-f6c5-4f46-9264-59f528fc57e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:21:19.030491 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:19.030453    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg" podUID="5191a2c0-f6c5-4f46-9264-59f528fc57e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:21:29.031304 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:29.031275    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg"
Sep 17 18:21:37.431327 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:37.431289    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg"]
Sep 17 18:21:37.431798 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:37.431548    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg" podUID="5191a2c0-f6c5-4f46-9264-59f528fc57e6" containerName="kserve-container" containerID="cri-o://da0661a59c926c40751bd5e674763afeb85095fc98a3ffd8d1104731b9cceccc" gracePeriod=30
Sep 17 18:21:37.431798 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:37.431593    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg" podUID="5191a2c0-f6c5-4f46-9264-59f528fc57e6" containerName="kube-rbac-proxy" containerID="cri-o://5391adf66bf92f6be5669b582142983f78c4289d4f3b151727178b33ce133dc3" gracePeriod=30
Sep 17 18:21:38.359746 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:38.359713    2581 generic.go:356] "Generic (PLEG): container finished" podID="5191a2c0-f6c5-4f46-9264-59f528fc57e6" containerID="5391adf66bf92f6be5669b582142983f78c4289d4f3b151727178b33ce133dc3" exitCode=2
Sep 17 18:21:38.359912 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:38.359788    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg" event={"ID":"5191a2c0-f6c5-4f46-9264-59f528fc57e6","Type":"ContainerDied","Data":"5391adf66bf92f6be5669b582142983f78c4289d4f3b151727178b33ce133dc3"}
Sep 17 18:21:39.026058 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:39.026020    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg" podUID="5191a2c0-f6c5-4f46-9264-59f528fc57e6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 17 18:21:39.030292 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:39.030265    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg" podUID="5191a2c0-f6c5-4f46-9264-59f528fc57e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:21:41.662732 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:41.662708    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg"
Sep 17 18:21:41.799870 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:41.799788    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5191a2c0-f6c5-4f46-9264-59f528fc57e6-cabundle-cert\") pod \"5191a2c0-f6c5-4f46-9264-59f528fc57e6\" (UID: \"5191a2c0-f6c5-4f46-9264-59f528fc57e6\") "
Sep 17 18:21:41.799870 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:41.799852    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjtgt\" (UniqueName: \"kubernetes.io/projected/5191a2c0-f6c5-4f46-9264-59f528fc57e6-kube-api-access-wjtgt\") pod \"5191a2c0-f6c5-4f46-9264-59f528fc57e6\" (UID: \"5191a2c0-f6c5-4f46-9264-59f528fc57e6\") "
Sep 17 18:21:41.800098 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:41.799884    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5191a2c0-f6c5-4f46-9264-59f528fc57e6-kserve-provision-location\") pod \"5191a2c0-f6c5-4f46-9264-59f528fc57e6\" (UID: \"5191a2c0-f6c5-4f46-9264-59f528fc57e6\") "
Sep 17 18:21:41.800098 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:41.799945    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5191a2c0-f6c5-4f46-9264-59f528fc57e6-kube-rbac-proxy-sar-config\") pod \"5191a2c0-f6c5-4f46-9264-59f528fc57e6\" (UID: \"5191a2c0-f6c5-4f46-9264-59f528fc57e6\") "
Sep 17 18:21:41.800098 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:41.799987    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5191a2c0-f6c5-4f46-9264-59f528fc57e6-proxy-tls\") pod \"5191a2c0-f6c5-4f46-9264-59f528fc57e6\" (UID: \"5191a2c0-f6c5-4f46-9264-59f528fc57e6\") "
Sep 17 18:21:41.800302 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:41.800270    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5191a2c0-f6c5-4f46-9264-59f528fc57e6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5191a2c0-f6c5-4f46-9264-59f528fc57e6" (UID: "5191a2c0-f6c5-4f46-9264-59f528fc57e6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:21:41.800352 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:41.800282    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5191a2c0-f6c5-4f46-9264-59f528fc57e6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5191a2c0-f6c5-4f46-9264-59f528fc57e6" (UID: "5191a2c0-f6c5-4f46-9264-59f528fc57e6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:21:41.800390 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:41.800349    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5191a2c0-f6c5-4f46-9264-59f528fc57e6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5191a2c0-f6c5-4f46-9264-59f528fc57e6" (UID: "5191a2c0-f6c5-4f46-9264-59f528fc57e6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:21:41.802063 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:41.802042    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5191a2c0-f6c5-4f46-9264-59f528fc57e6-kube-api-access-wjtgt" (OuterVolumeSpecName: "kube-api-access-wjtgt") pod "5191a2c0-f6c5-4f46-9264-59f528fc57e6" (UID: "5191a2c0-f6c5-4f46-9264-59f528fc57e6"). InnerVolumeSpecName "kube-api-access-wjtgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:21:41.802137 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:41.802094    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5191a2c0-f6c5-4f46-9264-59f528fc57e6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5191a2c0-f6c5-4f46-9264-59f528fc57e6" (UID: "5191a2c0-f6c5-4f46-9264-59f528fc57e6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:21:41.901383 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:41.901356    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5191a2c0-f6c5-4f46-9264-59f528fc57e6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:21:41.901383 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:41.901380    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5191a2c0-f6c5-4f46-9264-59f528fc57e6-proxy-tls\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:21:41.901547 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:41.901389    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5191a2c0-f6c5-4f46-9264-59f528fc57e6-cabundle-cert\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:21:41.901547 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:41.901397    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjtgt\" (UniqueName: \"kubernetes.io/projected/5191a2c0-f6c5-4f46-9264-59f528fc57e6-kube-api-access-wjtgt\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:21:41.901547 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:41.901405    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5191a2c0-f6c5-4f46-9264-59f528fc57e6-kserve-provision-location\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:21:42.378293 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:42.378255    2581 generic.go:356] "Generic (PLEG): container finished" podID="5191a2c0-f6c5-4f46-9264-59f528fc57e6" containerID="da0661a59c926c40751bd5e674763afeb85095fc98a3ffd8d1104731b9cceccc" exitCode=0
Sep 17 18:21:42.378475 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:42.378320    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg" event={"ID":"5191a2c0-f6c5-4f46-9264-59f528fc57e6","Type":"ContainerDied","Data":"da0661a59c926c40751bd5e674763afeb85095fc98a3ffd8d1104731b9cceccc"}
Sep 17 18:21:42.378475 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:42.378370    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg" event={"ID":"5191a2c0-f6c5-4f46-9264-59f528fc57e6","Type":"ContainerDied","Data":"c0d457779a9e10ef9e769c8429cf0dd5bf56089846afc49e7007ce3a47a873d0"}
Sep 17 18:21:42.378475 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:42.378395    2581 scope.go:117] "RemoveContainer" containerID="5391adf66bf92f6be5669b582142983f78c4289d4f3b151727178b33ce133dc3"
Sep 17 18:21:42.378475 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:42.378320    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg"
Sep 17 18:21:42.388141 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:42.388082    2581 scope.go:117] "RemoveContainer" containerID="da0661a59c926c40751bd5e674763afeb85095fc98a3ffd8d1104731b9cceccc"
Sep 17 18:21:42.394992 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:42.394968    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg"]
Sep 17 18:21:42.396287 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:42.396269    2581 scope.go:117] "RemoveContainer" containerID="53447951aa544f36c94db313b1063536b42363078beb673530d23a2fc60d0dc2"
Sep 17 18:21:42.396338 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:42.396307    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-ldrxg"]
Sep 17 18:21:42.403253 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:42.403236    2581 scope.go:117] "RemoveContainer" containerID="5391adf66bf92f6be5669b582142983f78c4289d4f3b151727178b33ce133dc3"
Sep 17 18:21:42.403492 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:21:42.403478    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5391adf66bf92f6be5669b582142983f78c4289d4f3b151727178b33ce133dc3\": container with ID starting with 5391adf66bf92f6be5669b582142983f78c4289d4f3b151727178b33ce133dc3 not found: ID does not exist" containerID="5391adf66bf92f6be5669b582142983f78c4289d4f3b151727178b33ce133dc3"
Sep 17 18:21:42.403539 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:42.403504    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5391adf66bf92f6be5669b582142983f78c4289d4f3b151727178b33ce133dc3"} err="failed to get container status \"5391adf66bf92f6be5669b582142983f78c4289d4f3b151727178b33ce133dc3\": rpc error: code = NotFound desc = could not find container \"5391adf66bf92f6be5669b582142983f78c4289d4f3b151727178b33ce133dc3\": container with ID starting with 5391adf66bf92f6be5669b582142983f78c4289d4f3b151727178b33ce133dc3 not found: ID does not exist"
Sep 17 18:21:42.403539 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:42.403521    2581 scope.go:117] "RemoveContainer" containerID="da0661a59c926c40751bd5e674763afeb85095fc98a3ffd8d1104731b9cceccc"
Sep 17 18:21:42.403756 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:21:42.403737    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da0661a59c926c40751bd5e674763afeb85095fc98a3ffd8d1104731b9cceccc\": container with ID starting with da0661a59c926c40751bd5e674763afeb85095fc98a3ffd8d1104731b9cceccc not found: ID does not exist" containerID="da0661a59c926c40751bd5e674763afeb85095fc98a3ffd8d1104731b9cceccc"
Sep 17 18:21:42.403790 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:42.403763    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da0661a59c926c40751bd5e674763afeb85095fc98a3ffd8d1104731b9cceccc"} err="failed to get container status \"da0661a59c926c40751bd5e674763afeb85095fc98a3ffd8d1104731b9cceccc\": rpc error: code = NotFound desc = could not find container \"da0661a59c926c40751bd5e674763afeb85095fc98a3ffd8d1104731b9cceccc\": container with ID starting with da0661a59c926c40751bd5e674763afeb85095fc98a3ffd8d1104731b9cceccc not found: ID does not exist"
Sep 17 18:21:42.403790 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:42.403782    2581 scope.go:117] "RemoveContainer" containerID="53447951aa544f36c94db313b1063536b42363078beb673530d23a2fc60d0dc2"
Sep 17 18:21:42.404024 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:21:42.404006    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53447951aa544f36c94db313b1063536b42363078beb673530d23a2fc60d0dc2\": container with ID starting with 53447951aa544f36c94db313b1063536b42363078beb673530d23a2fc60d0dc2 not found: ID does not exist" containerID="53447951aa544f36c94db313b1063536b42363078beb673530d23a2fc60d0dc2"
Sep 17 18:21:42.404086 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:42.404032    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53447951aa544f36c94db313b1063536b42363078beb673530d23a2fc60d0dc2"} err="failed to get container status \"53447951aa544f36c94db313b1063536b42363078beb673530d23a2fc60d0dc2\": rpc error: code = NotFound desc = could not find container \"53447951aa544f36c94db313b1063536b42363078beb673530d23a2fc60d0dc2\": container with ID starting with 53447951aa544f36c94db313b1063536b42363078beb673530d23a2fc60d0dc2 not found: ID does not exist"
Sep 17 18:21:44.048621 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:21:44.048590    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5191a2c0-f6c5-4f46-9264-59f528fc57e6" path="/var/lib/kubelet/pods/5191a2c0-f6c5-4f46-9264-59f528fc57e6/volumes"
Sep 17 18:22:00.015590 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:00.015556    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm8hx_913ce1e9-9fb6-4049-a06d-9ebc69d3e528/ovn-acl-logging/0.log"
Sep 17 18:22:00.017986 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:00.017966    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm8hx_913ce1e9-9fb6-4049-a06d-9ebc69d3e528/ovn-acl-logging/0.log"
Sep 17 18:22:01.848082 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.848049    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"]
Sep 17 18:22:01.848454 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.848352    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3" containerName="kube-rbac-proxy"
Sep 17 18:22:01.848454 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.848363    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3" containerName="kube-rbac-proxy"
Sep 17 18:22:01.848454 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.848374    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5191a2c0-f6c5-4f46-9264-59f528fc57e6" containerName="kserve-container"
Sep 17 18:22:01.848454 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.848379    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5191a2c0-f6c5-4f46-9264-59f528fc57e6" containerName="kserve-container"
Sep 17 18:22:01.848454 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.848385    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5191a2c0-f6c5-4f46-9264-59f528fc57e6" containerName="kube-rbac-proxy"
Sep 17 18:22:01.848454 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.848390    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5191a2c0-f6c5-4f46-9264-59f528fc57e6" containerName="kube-rbac-proxy"
Sep 17 18:22:01.848454 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.848404    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3" containerName="kserve-container"
Sep 17 18:22:01.848454 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.848410    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3" containerName="kserve-container"
Sep 17 18:22:01.848454 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.848418    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3" containerName="storage-initializer"
Sep 17 18:22:01.848454 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.848423    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3" containerName="storage-initializer"
Sep 17 18:22:01.848454 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.848433    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5191a2c0-f6c5-4f46-9264-59f528fc57e6" containerName="storage-initializer"
Sep 17 18:22:01.848454 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.848438    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5191a2c0-f6c5-4f46-9264-59f528fc57e6" containerName="storage-initializer"
Sep 17 18:22:01.848823 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.848482    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5191a2c0-f6c5-4f46-9264-59f528fc57e6" containerName="kube-rbac-proxy"
Sep 17 18:22:01.848823 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.848489    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5191a2c0-f6c5-4f46-9264-59f528fc57e6" containerName="kserve-container"
Sep 17 18:22:01.848823 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.848496    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3" containerName="kube-rbac-proxy"
Sep 17 18:22:01.848823 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.848501    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e6a2de7-9e3c-4387-9ec4-127ed99cb4d3" containerName="kserve-container"
Sep 17 18:22:01.852896 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.852878    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"
Sep 17 18:22:01.855224 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.855203    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\""
Sep 17 18:22:01.855389 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.855348    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:22:01.855389 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.855362    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w6q8k\""
Sep 17 18:22:01.855528 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.855415    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 17 18:22:01.855528 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.855419    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:22:01.855528 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.855357    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:22:01.855528 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.855520    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:22:01.861914 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.861895    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"]
Sep 17 18:22:01.949813 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.949792    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/071e5b27-e983-4fb7-b47d-72c409bdaaef-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7\" (UID: \"071e5b27-e983-4fb7-b47d-72c409bdaaef\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"
Sep 17 18:22:01.949947 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.949818    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4s6b\" (UniqueName: \"kubernetes.io/projected/071e5b27-e983-4fb7-b47d-72c409bdaaef-kube-api-access-m4s6b\") pod \"isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7\" (UID: \"071e5b27-e983-4fb7-b47d-72c409bdaaef\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"
Sep 17 18:22:01.949947 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.949849    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/071e5b27-e983-4fb7-b47d-72c409bdaaef-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7\" (UID: \"071e5b27-e983-4fb7-b47d-72c409bdaaef\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"
Sep 17 18:22:01.949947 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.949874    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/071e5b27-e983-4fb7-b47d-72c409bdaaef-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7\" (UID: \"071e5b27-e983-4fb7-b47d-72c409bdaaef\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"
Sep 17 18:22:01.949947 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:01.949923    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/071e5b27-e983-4fb7-b47d-72c409bdaaef-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7\" (UID: \"071e5b27-e983-4fb7-b47d-72c409bdaaef\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"
Sep 17 18:22:02.050454 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:02.050428    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/071e5b27-e983-4fb7-b47d-72c409bdaaef-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7\" (UID: \"071e5b27-e983-4fb7-b47d-72c409bdaaef\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"
Sep 17 18:22:02.050580 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:02.050468    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/071e5b27-e983-4fb7-b47d-72c409bdaaef-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7\" (UID: \"071e5b27-e983-4fb7-b47d-72c409bdaaef\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"
Sep 17 18:22:02.050580 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:02.050486    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/071e5b27-e983-4fb7-b47d-72c409bdaaef-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7\" (UID: \"071e5b27-e983-4fb7-b47d-72c409bdaaef\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"
Sep 17 18:22:02.050765 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:02.050746    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/071e5b27-e983-4fb7-b47d-72c409bdaaef-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7\" (UID: \"071e5b27-e983-4fb7-b47d-72c409bdaaef\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"
Sep 17 18:22:02.050836 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:02.050777    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4s6b\" (UniqueName: \"kubernetes.io/projected/071e5b27-e983-4fb7-b47d-72c409bdaaef-kube-api-access-m4s6b\") pod \"isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7\" (UID: \"071e5b27-e983-4fb7-b47d-72c409bdaaef\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"
Sep 17 18:22:02.051129 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:02.051108    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/071e5b27-e983-4fb7-b47d-72c409bdaaef-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7\" (UID: \"071e5b27-e983-4fb7-b47d-72c409bdaaef\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"
Sep 17 18:22:02.051267 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:02.051249    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/071e5b27-e983-4fb7-b47d-72c409bdaaef-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7\" (UID: \"071e5b27-e983-4fb7-b47d-72c409bdaaef\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"
Sep 17 18:22:02.051374 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:02.051358    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/071e5b27-e983-4fb7-b47d-72c409bdaaef-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7\" (UID: \"071e5b27-e983-4fb7-b47d-72c409bdaaef\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"
Sep 17 18:22:02.053070 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:02.053054    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/071e5b27-e983-4fb7-b47d-72c409bdaaef-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7\" (UID: \"071e5b27-e983-4fb7-b47d-72c409bdaaef\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"
Sep 17 18:22:02.059527 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:02.059509    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4s6b\" (UniqueName: \"kubernetes.io/projected/071e5b27-e983-4fb7-b47d-72c409bdaaef-kube-api-access-m4s6b\") pod \"isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7\" (UID: \"071e5b27-e983-4fb7-b47d-72c409bdaaef\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"
Sep 17 18:22:02.164415 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:02.164387    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"
Sep 17 18:22:02.437801 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:02.437778    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"]
Sep 17 18:22:02.440256 ip-10-0-134-209 kubenswrapper[2581]: W0917 18:22:02.440220    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod071e5b27_e983_4fb7_b47d_72c409bdaaef.slice/crio-2bf82369ca27138f5847326ecfecaba286077df3461b52c33735e6e19f574d37 WatchSource:0}: Error finding container 2bf82369ca27138f5847326ecfecaba286077df3461b52c33735e6e19f574d37: Status 404 returned error can't find the container with id 2bf82369ca27138f5847326ecfecaba286077df3461b52c33735e6e19f574d37
Sep 17 18:22:02.457959 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:02.457934    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" event={"ID":"071e5b27-e983-4fb7-b47d-72c409bdaaef","Type":"ContainerStarted","Data":"2bf82369ca27138f5847326ecfecaba286077df3461b52c33735e6e19f574d37"}
Sep 17 18:22:03.463489 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:03.463454    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" event={"ID":"071e5b27-e983-4fb7-b47d-72c409bdaaef","Type":"ContainerStarted","Data":"8ae5b8faadaea969c2cd0ee3c5529d622d4739b6e889181d5f506513eb658649"}
Sep 17 18:22:07.484547 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:07.484513    2581 generic.go:356] "Generic (PLEG): container finished" podID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerID="8ae5b8faadaea969c2cd0ee3c5529d622d4739b6e889181d5f506513eb658649" exitCode=0
Sep 17 18:22:07.484930 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:07.484585    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" event={"ID":"071e5b27-e983-4fb7-b47d-72c409bdaaef","Type":"ContainerDied","Data":"8ae5b8faadaea969c2cd0ee3c5529d622d4739b6e889181d5f506513eb658649"}
Sep 17 18:22:07.599046 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:07.599029    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:22:08.491953 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:08.491922    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" event={"ID":"071e5b27-e983-4fb7-b47d-72c409bdaaef","Type":"ContainerStarted","Data":"c82a61e980fe0087c3d590e0afe2543a0a1fbd9c96b4743ae2153728eacbf0ed"}
Sep 17 18:22:08.492314 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:08.491978    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" event={"ID":"071e5b27-e983-4fb7-b47d-72c409bdaaef","Type":"ContainerStarted","Data":"43630ee18b4d18c000b772e622ef1aa77b15889407431498078658454b7573bc"}
Sep 17 18:22:10.502721 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:10.502636    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" event={"ID":"071e5b27-e983-4fb7-b47d-72c409bdaaef","Type":"ContainerStarted","Data":"0b98a3bcac7251a25c388ed22d762190ffdd63c67e09b59a72dcaf6fb9d0c800"}
Sep 17 18:22:10.503120 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:10.502836    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"
Sep 17 18:22:10.503120 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:10.502929    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"
Sep 17 18:22:10.504032 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:10.504007    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 18:22:10.528466 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:10.528417    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podStartSLOduration=2.894302083 podStartE2EDuration="9.528399849s" totalImagesPullingTime="2.643596738s" totalInitContainerRuntime="3.990501028s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:22:10.52737395 +0000 UTC m=+1512.627837770" watchObservedRunningTime="2026-09-17 18:22:10.528399849 +0000 UTC m=+1512.628863668"
Sep 17 18:22:11.506682 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:11.506644    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"
Sep 17 18:22:11.506682 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:11.506651    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 18:22:11.507745 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:11.507719    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:22:12.510616 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:12.510566    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 18:22:12.511007 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:12.510977    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:22:12.516912 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:12.516893    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"
Sep 17 18:22:13.514836 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:13.514799    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 18:22:13.515225 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:13.515084    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:22:23.515633 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:23.515591    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 18:22:23.516211 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:23.516189    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:22:33.515088 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:33.515039    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 18:22:33.515627 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:33.515546    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:22:43.515380 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:43.515332    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 18:22:43.515940 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:43.515918    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:22:53.515357 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:53.515310    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 18:22:53.517725 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:22:53.515859    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:23:03.515549 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:03.515502    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 18:23:03.515938 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:03.515915    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:23:13.515674 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:13.515628    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 18:23:13.516230 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:13.516124    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:23:23.515804 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:23.515776    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"
Sep 17 18:23:23.516197 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:23.515939    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"
Sep 17 18:23:37.056955 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:37.056922    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"]
Sep 17 18:23:37.057359 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:37.057236    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="kserve-container" containerID="cri-o://43630ee18b4d18c000b772e622ef1aa77b15889407431498078658454b7573bc" gracePeriod=30
Sep 17 18:23:37.057359 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:37.057263    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="agent" containerID="cri-o://0b98a3bcac7251a25c388ed22d762190ffdd63c67e09b59a72dcaf6fb9d0c800" gracePeriod=30
Sep 17 18:23:37.057359 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:37.057322    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="kube-rbac-proxy" containerID="cri-o://c82a61e980fe0087c3d590e0afe2543a0a1fbd9c96b4743ae2153728eacbf0ed" gracePeriod=30
Sep 17 18:23:37.510998 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:37.510955    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Sep 17 18:23:37.850359 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:37.850274    2581 generic.go:356] "Generic (PLEG): container finished" podID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerID="c82a61e980fe0087c3d590e0afe2543a0a1fbd9c96b4743ae2153728eacbf0ed" exitCode=2
Sep 17 18:23:37.850359 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:37.850324    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" event={"ID":"071e5b27-e983-4fb7-b47d-72c409bdaaef","Type":"ContainerDied","Data":"c82a61e980fe0087c3d590e0afe2543a0a1fbd9c96b4743ae2153728eacbf0ed"}
Sep 17 18:23:41.868929 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:41.868896    2581 generic.go:356] "Generic (PLEG): container finished" podID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerID="43630ee18b4d18c000b772e622ef1aa77b15889407431498078658454b7573bc" exitCode=0
Sep 17 18:23:41.869324 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:41.868966    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" event={"ID":"071e5b27-e983-4fb7-b47d-72c409bdaaef","Type":"ContainerDied","Data":"43630ee18b4d18c000b772e622ef1aa77b15889407431498078658454b7573bc"}
Sep 17 18:23:42.511503 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:42.511461    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Sep 17 18:23:43.515421 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:43.515357    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 18:23:43.515802 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:43.515656    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:23:47.511074 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:47.511027    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Sep 17 18:23:47.511543 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:47.511188    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"
Sep 17 18:23:52.511469 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:52.511428    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Sep 17 18:23:53.515235 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:53.515192    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 18:23:53.515610 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:53.515514    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:23:53.873927 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:53.873878    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/kserve-container/0.log"
Sep 17 18:23:53.895817 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:53.895792    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/kube-rbac-proxy/0.log"
Sep 17 18:23:53.904845 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:53.904830    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/agent/0.log"
Sep 17 18:23:53.914255 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:53.914238    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/storage-initializer/0.log"
Sep 17 18:23:54.728523 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:54.728496    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/kserve-container/0.log"
Sep 17 18:23:54.739470 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:54.739450    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/kube-rbac-proxy/0.log"
Sep 17 18:23:54.749167 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:54.749123    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/agent/0.log"
Sep 17 18:23:54.758061 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:54.758043    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/storage-initializer/0.log"
Sep 17 18:23:55.570971 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:55.570945    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/kserve-container/0.log"
Sep 17 18:23:55.581722 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:55.581698    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/kube-rbac-proxy/0.log"
Sep 17 18:23:55.591514 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:55.591490    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/agent/0.log"
Sep 17 18:23:55.602355 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:55.602331    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/storage-initializer/0.log"
Sep 17 18:23:56.388847 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:56.388820    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/kserve-container/0.log"
Sep 17 18:23:56.402243 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:56.402223    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/kube-rbac-proxy/0.log"
Sep 17 18:23:56.411061 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:56.411040    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/agent/0.log"
Sep 17 18:23:56.420759 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:56.420738    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/storage-initializer/0.log"
Sep 17 18:23:57.206276 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:57.206247    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/kserve-container/0.log"
Sep 17 18:23:57.216443 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:57.216425    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/kube-rbac-proxy/0.log"
Sep 17 18:23:57.225654 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:57.225634    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/agent/0.log"
Sep 17 18:23:57.235542 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:57.235521    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/storage-initializer/0.log"
Sep 17 18:23:57.511103 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:57.511014    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Sep 17 18:23:58.042226 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:58.042123    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/kserve-container/0.log"
Sep 17 18:23:58.052971 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:58.052944    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/kube-rbac-proxy/0.log"
Sep 17 18:23:58.062415 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:58.062391    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/agent/0.log"
Sep 17 18:23:58.072924 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:58.072879    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/storage-initializer/0.log"
Sep 17 18:23:58.861459 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:58.861411    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/kserve-container/0.log"
Sep 17 18:23:58.871598 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:58.871579    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/kube-rbac-proxy/0.log"
Sep 17 18:23:58.881109 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:58.881088    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/agent/0.log"
Sep 17 18:23:58.890464 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:58.890436    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/storage-initializer/0.log"
Sep 17 18:23:59.678892 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:59.678868    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/kserve-container/0.log"
Sep 17 18:23:59.689429 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:59.689411    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/kube-rbac-proxy/0.log"
Sep 17 18:23:59.699192 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:59.699170    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/agent/0.log"
Sep 17 18:23:59.708766 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:23:59.708745    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/storage-initializer/0.log"
Sep 17 18:24:00.512702 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:00.512675    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/kserve-container/0.log"
Sep 17 18:24:00.525810 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:00.525779    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/kube-rbac-proxy/0.log"
Sep 17 18:24:00.538081 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:00.538066    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/agent/0.log"
Sep 17 18:24:00.548509 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:00.548477    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/storage-initializer/0.log"
Sep 17 18:24:01.342529 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:01.342496    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/kserve-container/0.log"
Sep 17 18:24:01.353502 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:01.353473    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/kube-rbac-proxy/0.log"
Sep 17 18:24:01.363250 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:01.363231    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/agent/0.log"
Sep 17 18:24:01.373240 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:01.373220    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/storage-initializer/0.log"
Sep 17 18:24:02.162502 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:02.162479    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/kserve-container/0.log"
Sep 17 18:24:02.172914 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:02.172893    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/kube-rbac-proxy/0.log"
Sep 17 18:24:02.182522 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:02.182504    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/agent/0.log"
Sep 17 18:24:02.192029 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:02.192012    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/storage-initializer/0.log"
Sep 17 18:24:02.511356 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:02.511270    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Sep 17 18:24:02.990585 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:02.990555    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/kserve-container/0.log"
Sep 17 18:24:03.000615 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:03.000589    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/kube-rbac-proxy/0.log"
Sep 17 18:24:03.009431 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:03.009407    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/agent/0.log"
Sep 17 18:24:03.019016 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:03.019001    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/storage-initializer/0.log"
Sep 17 18:24:03.514985 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:03.514947    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 18:24:03.515425 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:03.515118    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"
Sep 17 18:24:03.515425 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:03.515351    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:24:03.515504 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:03.515472    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"
Sep 17 18:24:03.852196 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:03.852101    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/kserve-container/0.log"
Sep 17 18:24:03.863228 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:03.863205    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/kube-rbac-proxy/0.log"
Sep 17 18:24:03.871953 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:03.871928    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/agent/0.log"
Sep 17 18:24:03.880970 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:03.880953    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/storage-initializer/0.log"
Sep 17 18:24:04.705638 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:04.705605    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/kserve-container/0.log"
Sep 17 18:24:04.715782 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:04.715742    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/kube-rbac-proxy/0.log"
Sep 17 18:24:04.725037 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:04.725016    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/agent/0.log"
Sep 17 18:24:04.733876 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:04.733859    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/storage-initializer/0.log"
Sep 17 18:24:05.577971 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:05.577933    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/kserve-container/0.log"
Sep 17 18:24:05.588350 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:05.588328    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/kube-rbac-proxy/0.log"
Sep 17 18:24:05.597523 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:05.597498    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/agent/0.log"
Sep 17 18:24:05.607268 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:05.607238    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7_071e5b27-e983-4fb7-b47d-72c409bdaaef/storage-initializer/0.log"
Sep 17 18:24:07.511556 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:07.511509    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Sep 17 18:24:07.693561 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:07.693538    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"
Sep 17 18:24:07.834325 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:07.834242    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/071e5b27-e983-4fb7-b47d-72c409bdaaef-kserve-provision-location\") pod \"071e5b27-e983-4fb7-b47d-72c409bdaaef\" (UID: \"071e5b27-e983-4fb7-b47d-72c409bdaaef\") "
Sep 17 18:24:07.834325 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:07.834297    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/071e5b27-e983-4fb7-b47d-72c409bdaaef-kube-rbac-proxy-sar-config\") pod \"071e5b27-e983-4fb7-b47d-72c409bdaaef\" (UID: \"071e5b27-e983-4fb7-b47d-72c409bdaaef\") "
Sep 17 18:24:07.834325 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:07.834327    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4s6b\" (UniqueName: \"kubernetes.io/projected/071e5b27-e983-4fb7-b47d-72c409bdaaef-kube-api-access-m4s6b\") pod \"071e5b27-e983-4fb7-b47d-72c409bdaaef\" (UID: \"071e5b27-e983-4fb7-b47d-72c409bdaaef\") "
Sep 17 18:24:07.834618 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:07.834376    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/071e5b27-e983-4fb7-b47d-72c409bdaaef-proxy-tls\") pod \"071e5b27-e983-4fb7-b47d-72c409bdaaef\" (UID: \"071e5b27-e983-4fb7-b47d-72c409bdaaef\") "
Sep 17 18:24:07.834618 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:07.834401    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/071e5b27-e983-4fb7-b47d-72c409bdaaef-cabundle-cert\") pod \"071e5b27-e983-4fb7-b47d-72c409bdaaef\" (UID: \"071e5b27-e983-4fb7-b47d-72c409bdaaef\") "
Sep 17 18:24:07.834618 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:07.834598    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/071e5b27-e983-4fb7-b47d-72c409bdaaef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "071e5b27-e983-4fb7-b47d-72c409bdaaef" (UID: "071e5b27-e983-4fb7-b47d-72c409bdaaef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:24:07.834759 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:07.834671    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/071e5b27-e983-4fb7-b47d-72c409bdaaef-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "071e5b27-e983-4fb7-b47d-72c409bdaaef" (UID: "071e5b27-e983-4fb7-b47d-72c409bdaaef"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:24:07.834796 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:07.834777    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/071e5b27-e983-4fb7-b47d-72c409bdaaef-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "071e5b27-e983-4fb7-b47d-72c409bdaaef" (UID: "071e5b27-e983-4fb7-b47d-72c409bdaaef"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:24:07.836521 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:07.836501    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/071e5b27-e983-4fb7-b47d-72c409bdaaef-kube-api-access-m4s6b" (OuterVolumeSpecName: "kube-api-access-m4s6b") pod "071e5b27-e983-4fb7-b47d-72c409bdaaef" (UID: "071e5b27-e983-4fb7-b47d-72c409bdaaef"). InnerVolumeSpecName "kube-api-access-m4s6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:24:07.836582 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:07.836539    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/071e5b27-e983-4fb7-b47d-72c409bdaaef-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "071e5b27-e983-4fb7-b47d-72c409bdaaef" (UID: "071e5b27-e983-4fb7-b47d-72c409bdaaef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:24:07.935915 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:07.935880    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4s6b\" (UniqueName: \"kubernetes.io/projected/071e5b27-e983-4fb7-b47d-72c409bdaaef-kube-api-access-m4s6b\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:24:07.935915 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:07.935910    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/071e5b27-e983-4fb7-b47d-72c409bdaaef-proxy-tls\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:24:07.935915 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:07.935919    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/071e5b27-e983-4fb7-b47d-72c409bdaaef-cabundle-cert\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:24:07.936140 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:07.935927    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/071e5b27-e983-4fb7-b47d-72c409bdaaef-kserve-provision-location\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:24:07.936140 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:07.935936    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/071e5b27-e983-4fb7-b47d-72c409bdaaef-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-209.ec2.internal\" DevicePath \"\""
Sep 17 18:24:07.972288 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:07.972257    2581 generic.go:356] "Generic (PLEG): container finished" podID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerID="0b98a3bcac7251a25c388ed22d762190ffdd63c67e09b59a72dcaf6fb9d0c800" exitCode=0
Sep 17 18:24:07.972432 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:07.972300    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" event={"ID":"071e5b27-e983-4fb7-b47d-72c409bdaaef","Type":"ContainerDied","Data":"0b98a3bcac7251a25c388ed22d762190ffdd63c67e09b59a72dcaf6fb9d0c800"}
Sep 17 18:24:07.972432 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:07.972343    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"
Sep 17 18:24:07.972432 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:07.972389    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7" event={"ID":"071e5b27-e983-4fb7-b47d-72c409bdaaef","Type":"ContainerDied","Data":"2bf82369ca27138f5847326ecfecaba286077df3461b52c33735e6e19f574d37"}
Sep 17 18:24:07.972432 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:07.972420    2581 scope.go:117] "RemoveContainer" containerID="0b98a3bcac7251a25c388ed22d762190ffdd63c67e09b59a72dcaf6fb9d0c800"
Sep 17 18:24:07.980684 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:07.980669    2581 scope.go:117] "RemoveContainer" containerID="c82a61e980fe0087c3d590e0afe2543a0a1fbd9c96b4743ae2153728eacbf0ed"
Sep 17 18:24:07.987864 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:07.987847    2581 scope.go:117] "RemoveContainer" containerID="43630ee18b4d18c000b772e622ef1aa77b15889407431498078658454b7573bc"
Sep 17 18:24:07.992279 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:07.992257    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"]
Sep 17 18:24:07.995714 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:07.995641    2581 scope.go:117] "RemoveContainer" containerID="8ae5b8faadaea969c2cd0ee3c5529d622d4739b6e889181d5f506513eb658649"
Sep 17 18:24:07.996070 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:07.996052    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6d54d95b97-vmkp7"]
Sep 17 18:24:08.002659 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.002642    2581 scope.go:117] "RemoveContainer" containerID="0b98a3bcac7251a25c388ed22d762190ffdd63c67e09b59a72dcaf6fb9d0c800"
Sep 17 18:24:08.002900 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:24:08.002881    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b98a3bcac7251a25c388ed22d762190ffdd63c67e09b59a72dcaf6fb9d0c800\": container with ID starting with 0b98a3bcac7251a25c388ed22d762190ffdd63c67e09b59a72dcaf6fb9d0c800 not found: ID does not exist" containerID="0b98a3bcac7251a25c388ed22d762190ffdd63c67e09b59a72dcaf6fb9d0c800"
Sep 17 18:24:08.002970 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.002913    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b98a3bcac7251a25c388ed22d762190ffdd63c67e09b59a72dcaf6fb9d0c800"} err="failed to get container status \"0b98a3bcac7251a25c388ed22d762190ffdd63c67e09b59a72dcaf6fb9d0c800\": rpc error: code = NotFound desc = could not find container \"0b98a3bcac7251a25c388ed22d762190ffdd63c67e09b59a72dcaf6fb9d0c800\": container with ID starting with 0b98a3bcac7251a25c388ed22d762190ffdd63c67e09b59a72dcaf6fb9d0c800 not found: ID does not exist"
Sep 17 18:24:08.002970 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.002938    2581 scope.go:117] "RemoveContainer" containerID="c82a61e980fe0087c3d590e0afe2543a0a1fbd9c96b4743ae2153728eacbf0ed"
Sep 17 18:24:08.003230 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:24:08.003213    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c82a61e980fe0087c3d590e0afe2543a0a1fbd9c96b4743ae2153728eacbf0ed\": container with ID starting with c82a61e980fe0087c3d590e0afe2543a0a1fbd9c96b4743ae2153728eacbf0ed not found: ID does not exist" containerID="c82a61e980fe0087c3d590e0afe2543a0a1fbd9c96b4743ae2153728eacbf0ed"
Sep 17 18:24:08.003276 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.003236    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c82a61e980fe0087c3d590e0afe2543a0a1fbd9c96b4743ae2153728eacbf0ed"} err="failed to get container status \"c82a61e980fe0087c3d590e0afe2543a0a1fbd9c96b4743ae2153728eacbf0ed\": rpc error: code = NotFound desc = could not find container \"c82a61e980fe0087c3d590e0afe2543a0a1fbd9c96b4743ae2153728eacbf0ed\": container with ID starting with c82a61e980fe0087c3d590e0afe2543a0a1fbd9c96b4743ae2153728eacbf0ed not found: ID does not exist"
Sep 17 18:24:08.003276 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.003251    2581 scope.go:117] "RemoveContainer" containerID="43630ee18b4d18c000b772e622ef1aa77b15889407431498078658454b7573bc"
Sep 17 18:24:08.003464 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:24:08.003450    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43630ee18b4d18c000b772e622ef1aa77b15889407431498078658454b7573bc\": container with ID starting with 43630ee18b4d18c000b772e622ef1aa77b15889407431498078658454b7573bc not found: ID does not exist" containerID="43630ee18b4d18c000b772e622ef1aa77b15889407431498078658454b7573bc"
Sep 17 18:24:08.003513 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.003467    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43630ee18b4d18c000b772e622ef1aa77b15889407431498078658454b7573bc"} err="failed to get container status \"43630ee18b4d18c000b772e622ef1aa77b15889407431498078658454b7573bc\": rpc error: code = NotFound desc = could not find container \"43630ee18b4d18c000b772e622ef1aa77b15889407431498078658454b7573bc\": container with ID starting with 43630ee18b4d18c000b772e622ef1aa77b15889407431498078658454b7573bc not found: ID does not exist"
Sep 17 18:24:08.003513 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.003479    2581 scope.go:117] "RemoveContainer" containerID="8ae5b8faadaea969c2cd0ee3c5529d622d4739b6e889181d5f506513eb658649"
Sep 17 18:24:08.003705 ip-10-0-134-209 kubenswrapper[2581]: E0917 18:24:08.003688    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ae5b8faadaea969c2cd0ee3c5529d622d4739b6e889181d5f506513eb658649\": container with ID starting with 8ae5b8faadaea969c2cd0ee3c5529d622d4739b6e889181d5f506513eb658649 not found: ID does not exist" containerID="8ae5b8faadaea969c2cd0ee3c5529d622d4739b6e889181d5f506513eb658649"
Sep 17 18:24:08.003762 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.003712    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ae5b8faadaea969c2cd0ee3c5529d622d4739b6e889181d5f506513eb658649"} err="failed to get container status \"8ae5b8faadaea969c2cd0ee3c5529d622d4739b6e889181d5f506513eb658649\": rpc error: code = NotFound desc = could not find container \"8ae5b8faadaea969c2cd0ee3c5529d622d4739b6e889181d5f506513eb658649\": container with ID starting with 8ae5b8faadaea969c2cd0ee3c5529d622d4739b6e889181d5f506513eb658649 not found: ID does not exist"
Sep 17 18:24:08.048639 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.048606    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" path="/var/lib/kubelet/pods/071e5b27-e983-4fb7-b47d-72c409bdaaef/volumes"
Sep 17 18:24:08.262311 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.262285    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rcdkz/must-gather-m4jlc"]
Sep 17 18:24:08.262560 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.262548    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="kserve-container"
Sep 17 18:24:08.262602 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.262561    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="kserve-container"
Sep 17 18:24:08.262602 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.262580    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="storage-initializer"
Sep 17 18:24:08.262602 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.262586    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="storage-initializer"
Sep 17 18:24:08.262602 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.262593    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="agent"
Sep 17 18:24:08.262602 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.262598    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="agent"
Sep 17 18:24:08.262602 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.262603    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="kube-rbac-proxy"
Sep 17 18:24:08.262774 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.262608    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="kube-rbac-proxy"
Sep 17 18:24:08.262774 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.262658    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="kube-rbac-proxy"
Sep 17 18:24:08.262774 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.262667    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="kserve-container"
Sep 17 18:24:08.262774 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.262672    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="071e5b27-e983-4fb7-b47d-72c409bdaaef" containerName="agent"
Sep 17 18:24:08.266844 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.266827    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rcdkz/must-gather-m4jlc"
Sep 17 18:24:08.269226 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.269192    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rcdkz\"/\"openshift-service-ca.crt\""
Sep 17 18:24:08.269335 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.269267    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rcdkz\"/\"default-dockercfg-667zq\""
Sep 17 18:24:08.269398 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.269338    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rcdkz\"/\"kube-root-ca.crt\""
Sep 17 18:24:08.273114 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.273093    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rcdkz/must-gather-m4jlc"]
Sep 17 18:24:08.339169 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.339121    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/71a47730-2330-4ee6-83cd-182899987a47-must-gather-output\") pod \"must-gather-m4jlc\" (UID: \"71a47730-2330-4ee6-83cd-182899987a47\") " pod="openshift-must-gather-rcdkz/must-gather-m4jlc"
Sep 17 18:24:08.339278 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.339220    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8c8n\" (UniqueName: \"kubernetes.io/projected/71a47730-2330-4ee6-83cd-182899987a47-kube-api-access-p8c8n\") pod \"must-gather-m4jlc\" (UID: \"71a47730-2330-4ee6-83cd-182899987a47\") " pod="openshift-must-gather-rcdkz/must-gather-m4jlc"
Sep 17 18:24:08.440070 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.440039    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/71a47730-2330-4ee6-83cd-182899987a47-must-gather-output\") pod \"must-gather-m4jlc\" (UID: \"71a47730-2330-4ee6-83cd-182899987a47\") " pod="openshift-must-gather-rcdkz/must-gather-m4jlc"
Sep 17 18:24:08.440217 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.440080    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8c8n\" (UniqueName: \"kubernetes.io/projected/71a47730-2330-4ee6-83cd-182899987a47-kube-api-access-p8c8n\") pod \"must-gather-m4jlc\" (UID: \"71a47730-2330-4ee6-83cd-182899987a47\") " pod="openshift-must-gather-rcdkz/must-gather-m4jlc"
Sep 17 18:24:08.440418 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.440399    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/71a47730-2330-4ee6-83cd-182899987a47-must-gather-output\") pod \"must-gather-m4jlc\" (UID: \"71a47730-2330-4ee6-83cd-182899987a47\") " pod="openshift-must-gather-rcdkz/must-gather-m4jlc"
Sep 17 18:24:08.448979 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.448948    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8c8n\" (UniqueName: \"kubernetes.io/projected/71a47730-2330-4ee6-83cd-182899987a47-kube-api-access-p8c8n\") pod \"must-gather-m4jlc\" (UID: \"71a47730-2330-4ee6-83cd-182899987a47\") " pod="openshift-must-gather-rcdkz/must-gather-m4jlc"
Sep 17 18:24:08.588412 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.588324    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rcdkz/must-gather-m4jlc"
Sep 17 18:24:08.863908 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.863847    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rcdkz/must-gather-m4jlc"]
Sep 17 18:24:08.866694 ip-10-0-134-209 kubenswrapper[2581]: W0917 18:24:08.866669    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71a47730_2330_4ee6_83cd_182899987a47.slice/crio-48715a8e2d68bc42fa7092d9c369dd8218c396d5af1fd8c77c16111e2fdcf21c WatchSource:0}: Error finding container 48715a8e2d68bc42fa7092d9c369dd8218c396d5af1fd8c77c16111e2fdcf21c: Status 404 returned error can't find the container with id 48715a8e2d68bc42fa7092d9c369dd8218c396d5af1fd8c77c16111e2fdcf21c
Sep 17 18:24:08.977173 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:08.977117    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rcdkz/must-gather-m4jlc" event={"ID":"71a47730-2330-4ee6-83cd-182899987a47","Type":"ContainerStarted","Data":"48715a8e2d68bc42fa7092d9c369dd8218c396d5af1fd8c77c16111e2fdcf21c"}
Sep 17 18:24:10.989509 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:10.989469    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rcdkz/must-gather-m4jlc" event={"ID":"71a47730-2330-4ee6-83cd-182899987a47","Type":"ContainerStarted","Data":"46c144d90d754c3c9cb86dccdcd54fe049aeea66fb98f67992287c6874644446"}
Sep 17 18:24:10.989904 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:10.989528    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rcdkz/must-gather-m4jlc" event={"ID":"71a47730-2330-4ee6-83cd-182899987a47","Type":"ContainerStarted","Data":"6efee5e3525d696b6bd9c70e4c393290c00ccab5db18606af8e53854dacb9b92"}
Sep 17 18:24:11.004413 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:11.004350    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rcdkz/must-gather-m4jlc" podStartSLOduration=1.801355707 podStartE2EDuration="3.004329895s" totalImagesPullingTime="1.202974188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:24:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:24:11.004049523 +0000 UTC m=+1633.104513343" watchObservedRunningTime="2026-09-17 18:24:11.004329895 +0000 UTC m=+1633.104793715"
Sep 17 18:24:11.557357 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:11.557323    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-shvtg_41ab02fa-694e-4b41-8969-0e5181664b3a/global-pull-secret-syncer/0.log"
Sep 17 18:24:11.671027 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:11.670993    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sf7g4_ca53e15c-51d9-40c8-9d87-80e217ce1815/konnectivity-agent/0.log"
Sep 17 18:24:11.691355 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:11.691271    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-209.ec2.internal_284822b4778f9378098f5e86308878b3/haproxy/0.log"
Sep 17 18:24:15.932719 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:15.932621    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-jxspn_b694aae6-aaf3-4091-9466-e336206bfc7e/kube-state-metrics/0.log"
Sep 17 18:24:15.956624 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:15.956591    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-jxspn_b694aae6-aaf3-4091-9466-e336206bfc7e/kube-rbac-proxy-main/0.log"
Sep 17 18:24:15.982569 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:15.982507    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-jxspn_b694aae6-aaf3-4091-9466-e336206bfc7e/kube-rbac-proxy-self/0.log"
Sep 17 18:24:16.022175 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:16.022108    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7644695647-fhdhz_5a2f9d6f-5315-40e6-9339-356a71997283/metrics-server/0.log"
Sep 17 18:24:16.078003 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:16.077964    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2mr86_ebeafe22-7486-457b-95cc-d9fd0d4496cf/node-exporter/0.log"
Sep 17 18:24:16.102677 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:16.102651    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2mr86_ebeafe22-7486-457b-95cc-d9fd0d4496cf/kube-rbac-proxy/0.log"
Sep 17 18:24:16.125880 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:16.125847    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2mr86_ebeafe22-7486-457b-95cc-d9fd0d4496cf/init-textfile/0.log"
Sep 17 18:24:16.381920 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:16.381891    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d/prometheus/0.log"
Sep 17 18:24:16.398653 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:16.398619    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d/config-reloader/0.log"
Sep 17 18:24:16.420886 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:16.420857    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d/thanos-sidecar/0.log"
Sep 17 18:24:16.444950 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:16.444914    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d/kube-rbac-proxy-web/0.log"
Sep 17 18:24:16.467511 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:16.467441    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d/kube-rbac-proxy/0.log"
Sep 17 18:24:16.489343 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:16.489307    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d/kube-rbac-proxy-thanos/0.log"
Sep 17 18:24:16.515647 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:16.515610    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_43a7f567-0cf3-4f0f-90f4-b5cdbb057a4d/init-config-reloader/0.log"
Sep 17 18:24:16.622410 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:16.622386    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b9cffb6c-cf2ws_85a6db98-a429-4f14-8d1e-98ed401382d9/telemeter-client/0.log"
Sep 17 18:24:16.654700 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:16.654626    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b9cffb6c-cf2ws_85a6db98-a429-4f14-8d1e-98ed401382d9/reload/0.log"
Sep 17 18:24:16.677918 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:16.677894    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b9cffb6c-cf2ws_85a6db98-a429-4f14-8d1e-98ed401382d9/kube-rbac-proxy/0.log"
Sep 17 18:24:18.176300 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:18.176274    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-2pp8d_edd2cec8-35e2-4f8d-97db-99de69508531/networking-console-plugin/0.log"
Sep 17 18:24:19.311227 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:19.311191    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rcdkz/perf-node-gather-daemonset-mk2zh"]
Sep 17 18:24:19.316126 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:19.316101    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-mk2zh"
Sep 17 18:24:19.343503 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:19.343469    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rcdkz/perf-node-gather-daemonset-mk2zh"]
Sep 17 18:24:19.345693 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:19.345660    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlrc8\" (UniqueName: \"kubernetes.io/projected/d9495877-5af7-40ec-8bda-d98c5b15a04c-kube-api-access-zlrc8\") pod \"perf-node-gather-daemonset-mk2zh\" (UID: \"d9495877-5af7-40ec-8bda-d98c5b15a04c\") " pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-mk2zh"
Sep 17 18:24:19.345846 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:19.345725    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d9495877-5af7-40ec-8bda-d98c5b15a04c-proc\") pod \"perf-node-gather-daemonset-mk2zh\" (UID: \"d9495877-5af7-40ec-8bda-d98c5b15a04c\") " pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-mk2zh"
Sep 17 18:24:19.345846 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:19.345777    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d9495877-5af7-40ec-8bda-d98c5b15a04c-podres\") pod \"perf-node-gather-daemonset-mk2zh\" (UID: \"d9495877-5af7-40ec-8bda-d98c5b15a04c\") " pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-mk2zh"
Sep 17 18:24:19.345846 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:19.345836    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9495877-5af7-40ec-8bda-d98c5b15a04c-lib-modules\") pod \"perf-node-gather-daemonset-mk2zh\" (UID: \"d9495877-5af7-40ec-8bda-d98c5b15a04c\") " pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-mk2zh"
Sep 17 18:24:19.346012 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:19.345885    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9495877-5af7-40ec-8bda-d98c5b15a04c-sys\") pod \"perf-node-gather-daemonset-mk2zh\" (UID: \"d9495877-5af7-40ec-8bda-d98c5b15a04c\") " pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-mk2zh"
Sep 17 18:24:19.447306 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:19.447269    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d9495877-5af7-40ec-8bda-d98c5b15a04c-podres\") pod \"perf-node-gather-daemonset-mk2zh\" (UID: \"d9495877-5af7-40ec-8bda-d98c5b15a04c\") " pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-mk2zh"
Sep 17 18:24:19.447488 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:19.447324    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9495877-5af7-40ec-8bda-d98c5b15a04c-lib-modules\") pod \"perf-node-gather-daemonset-mk2zh\" (UID: \"d9495877-5af7-40ec-8bda-d98c5b15a04c\") " pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-mk2zh"
Sep 17 18:24:19.447488 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:19.447364    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9495877-5af7-40ec-8bda-d98c5b15a04c-sys\") pod \"perf-node-gather-daemonset-mk2zh\" (UID: \"d9495877-5af7-40ec-8bda-d98c5b15a04c\") " pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-mk2zh"
Sep 17 18:24:19.447488 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:19.447408    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlrc8\" (UniqueName: \"kubernetes.io/projected/d9495877-5af7-40ec-8bda-d98c5b15a04c-kube-api-access-zlrc8\") pod \"perf-node-gather-daemonset-mk2zh\" (UID: \"d9495877-5af7-40ec-8bda-d98c5b15a04c\") " pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-mk2zh"
Sep 17 18:24:19.447488 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:19.447435    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d9495877-5af7-40ec-8bda-d98c5b15a04c-proc\") pod \"perf-node-gather-daemonset-mk2zh\" (UID: \"d9495877-5af7-40ec-8bda-d98c5b15a04c\") " pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-mk2zh"
Sep 17 18:24:19.447697 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:19.447487    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9495877-5af7-40ec-8bda-d98c5b15a04c-sys\") pod \"perf-node-gather-daemonset-mk2zh\" (UID: \"d9495877-5af7-40ec-8bda-d98c5b15a04c\") " pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-mk2zh"
Sep 17 18:24:19.447697 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:19.447503    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d9495877-5af7-40ec-8bda-d98c5b15a04c-proc\") pod \"perf-node-gather-daemonset-mk2zh\" (UID: \"d9495877-5af7-40ec-8bda-d98c5b15a04c\") " pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-mk2zh"
Sep 17 18:24:19.447697 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:19.447451    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d9495877-5af7-40ec-8bda-d98c5b15a04c-podres\") pod \"perf-node-gather-daemonset-mk2zh\" (UID: \"d9495877-5af7-40ec-8bda-d98c5b15a04c\") " pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-mk2zh"
Sep 17 18:24:19.447697 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:19.447537    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9495877-5af7-40ec-8bda-d98c5b15a04c-lib-modules\") pod \"perf-node-gather-daemonset-mk2zh\" (UID: \"d9495877-5af7-40ec-8bda-d98c5b15a04c\") " pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-mk2zh"
Sep 17 18:24:19.457116 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:19.457082    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlrc8\" (UniqueName: \"kubernetes.io/projected/d9495877-5af7-40ec-8bda-d98c5b15a04c-kube-api-access-zlrc8\") pod \"perf-node-gather-daemonset-mk2zh\" (UID: \"d9495877-5af7-40ec-8bda-d98c5b15a04c\") " pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-mk2zh"
Sep 17 18:24:19.627828 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:19.627786    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-mk2zh"
Sep 17 18:24:19.843470 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:19.843413    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-rnprf_36a45fc1-bf32-4f4b-b741-75ac26197973/volume-data-source-validator/0.log"
Sep 17 18:24:19.922621 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:19.922540    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rcdkz/perf-node-gather-daemonset-mk2zh"]
Sep 17 18:24:19.928874 ip-10-0-134-209 kubenswrapper[2581]: W0917 18:24:19.928832    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd9495877_5af7_40ec_8bda_d98c5b15a04c.slice/crio-b940cc54b7c3e8f8f9f6dd178b95e5120ad0caedfcfbc3fed23c37e3f2197af0 WatchSource:0}: Error finding container b940cc54b7c3e8f8f9f6dd178b95e5120ad0caedfcfbc3fed23c37e3f2197af0: Status 404 returned error can't find the container with id b940cc54b7c3e8f8f9f6dd178b95e5120ad0caedfcfbc3fed23c37e3f2197af0
Sep 17 18:24:20.050953 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:20.050921    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-mk2zh"
Sep 17 18:24:20.051120 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:20.050983    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-mk2zh" event={"ID":"d9495877-5af7-40ec-8bda-d98c5b15a04c","Type":"ContainerStarted","Data":"58945fb6cc248e677f695a422d6d502dfa55608deaacf77a5a190284800202b3"}
Sep 17 18:24:20.051120 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:20.051013    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-mk2zh" event={"ID":"d9495877-5af7-40ec-8bda-d98c5b15a04c","Type":"ContainerStarted","Data":"b940cc54b7c3e8f8f9f6dd178b95e5120ad0caedfcfbc3fed23c37e3f2197af0"}
Sep 17 18:24:20.070122 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:20.070070    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-mk2zh" podStartSLOduration=1.070050384 podStartE2EDuration="1.070050384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:24:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:24:20.068178851 +0000 UTC m=+1642.168642667" watchObservedRunningTime="2026-09-17 18:24:20.070050384 +0000 UTC m=+1642.170514203"
Sep 17 18:24:20.739269 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:20.739242    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jhhsj_a1d64f49-582d-4389-b190-b84fae699d42/dns/0.log"
Sep 17 18:24:20.758844 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:20.758818    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jhhsj_a1d64f49-582d-4389-b190-b84fae699d42/kube-rbac-proxy/0.log"
Sep 17 18:24:20.875421 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:20.875391    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2wq8v_6345bbfb-99aa-42ba-b5c9-d7c46ef2392f/dns-node-resolver/0.log"
Sep 17 18:24:21.351497 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:21.351472    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-56d8969c4d-qgjx4_3ad48dd9-2592-40af-822e-8b71a26febfc/registry/0.log"
Sep 17 18:24:21.413897 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:21.413871    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k7mgr_383b592e-5a46-4299-9522-ac835c0adc61/node-ca/0.log"
Sep 17 18:24:22.442655 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:22.442621    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-p7h8g_47da4adc-a0aa-4908-91b7-d3baae40714d/serve-healthcheck-canary/0.log"
Sep 17 18:24:22.830340 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:22.830276    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-ztc5j_1080ecc5-cca9-47db-b949-05bdd3777351/insights-operator/0.log"
Sep 17 18:24:22.830668 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:22.830641    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-ztc5j_1080ecc5-cca9-47db-b949-05bdd3777351/insights-operator/1.log"
Sep 17 18:24:22.996847 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:22.996818    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zdthk_6efbbf7f-2529-4207-9138-8ff135899614/kube-rbac-proxy/0.log"
Sep 17 18:24:23.039659 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:23.039633    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zdthk_6efbbf7f-2529-4207-9138-8ff135899614/exporter/0.log"
Sep 17 18:24:23.102200 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:23.102119    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zdthk_6efbbf7f-2529-4207-9138-8ff135899614/extractor/0.log"
Sep 17 18:24:25.587701 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:25.587675    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-xpktj_3f612f22-acc0-41bc-99e0-6db0a1eb887e/s3-init/0.log"
Sep 17 18:24:25.604011 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:25.603993    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-xpktj_3f612f22-acc0-41bc-99e0-6db0a1eb887e/download-hf-model/0.log"
Sep 17 18:24:26.063929 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:26.063903    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-mk2zh"
Sep 17 18:24:30.084204 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:30.084134    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-cfm6h_184b0189-e1d0-4bfd-b970-596373fd0745/kube-storage-version-migrator-operator/1.log"
Sep 17 18:24:30.084768 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:30.084744    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-cfm6h_184b0189-e1d0-4bfd-b970-596373fd0745/kube-storage-version-migrator-operator/0.log"
Sep 17 18:24:31.281012 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:31.280983    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8qdrq_38076ecb-73ae-4c1c-8359-b9e2b7435a7b/kube-multus-additional-cni-plugins/0.log"
Sep 17 18:24:31.300325 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:31.300278    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8qdrq_38076ecb-73ae-4c1c-8359-b9e2b7435a7b/egress-router-binary-copy/0.log"
Sep 17 18:24:31.323331 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:31.323302    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8qdrq_38076ecb-73ae-4c1c-8359-b9e2b7435a7b/cni-plugins/0.log"
Sep 17 18:24:31.343540 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:31.343518    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8qdrq_38076ecb-73ae-4c1c-8359-b9e2b7435a7b/bond-cni-plugin/0.log"
Sep 17 18:24:31.365451 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:31.365429    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8qdrq_38076ecb-73ae-4c1c-8359-b9e2b7435a7b/routeoverride-cni/0.log"
Sep 17 18:24:31.384941 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:31.384921    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8qdrq_38076ecb-73ae-4c1c-8359-b9e2b7435a7b/whereabouts-cni-bincopy/0.log"
Sep 17 18:24:31.403438 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:31.403416    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8qdrq_38076ecb-73ae-4c1c-8359-b9e2b7435a7b/whereabouts-cni/0.log"
Sep 17 18:24:31.504542 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:31.504514    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xhbcn_bd3ffd9f-338b-48ca-97c2-4292e8716182/kube-multus/0.log"
Sep 17 18:24:31.565789 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:31.565700    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dsz4t_11b6807f-cb26-4117-b06d-eb974e5640ae/network-metrics-daemon/0.log"
Sep 17 18:24:31.583101 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:31.583076    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dsz4t_11b6807f-cb26-4117-b06d-eb974e5640ae/kube-rbac-proxy/0.log"
Sep 17 18:24:32.568101 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:32.568052    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm8hx_913ce1e9-9fb6-4049-a06d-9ebc69d3e528/ovn-controller/0.log"
Sep 17 18:24:32.583441 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:32.583403    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm8hx_913ce1e9-9fb6-4049-a06d-9ebc69d3e528/ovn-acl-logging/1.log"
Sep 17 18:24:32.583613 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:32.583488    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm8hx_913ce1e9-9fb6-4049-a06d-9ebc69d3e528/ovn-acl-logging/0.log"
Sep 17 18:24:32.599920 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:32.599892    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm8hx_913ce1e9-9fb6-4049-a06d-9ebc69d3e528/kube-rbac-proxy-node/0.log"
Sep 17 18:24:32.619765 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:32.619689    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm8hx_913ce1e9-9fb6-4049-a06d-9ebc69d3e528/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 18:24:32.639535 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:32.639514    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm8hx_913ce1e9-9fb6-4049-a06d-9ebc69d3e528/northd/0.log"
Sep 17 18:24:32.657898 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:32.657882    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm8hx_913ce1e9-9fb6-4049-a06d-9ebc69d3e528/nbdb/0.log"
Sep 17 18:24:32.676320 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:32.676301    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm8hx_913ce1e9-9fb6-4049-a06d-9ebc69d3e528/sbdb/0.log"
Sep 17 18:24:32.717458 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:32.717433    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm8hx_913ce1e9-9fb6-4049-a06d-9ebc69d3e528/ovnkube-controller/0.log"
Sep 17 18:24:33.885514 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:33.885488    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7cbc45fddc-j2gss_1b091399-61b4-4444-a8c9-9d2849a7a2a9/check-endpoints/0.log"
Sep 17 18:24:33.909033 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:33.909010    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-98c4r_c53ddf27-1765-4785-b122-1645ccc8d094/network-check-target-container/0.log"
Sep 17 18:24:34.833880 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:34.833850    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-4qkk8_b4f2a63c-9017-431f-bec9-47521af4d1bd/iptables-alerter/0.log"
Sep 17 18:24:35.478719 ip-10-0-134-209 kubenswrapper[2581]: I0917 18:24:35.478645    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-mvn46_7ba4edf3-c791-4134-9ea3-449a5c3b437c/tuned/0.log"